Sep 14 10:46:37.972988 ip-10-0-136-89 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 14 10:46:37.973173 ip-10-0-136-89 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 14 10:46:37.973260 ip-10-0-136-89 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 14 10:46:37.973735 ip-10-0-136-89 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 14 10:46:46.890202 ip-10-0-136-89 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 14 10:46:46.890215 ip-10-0-136-89 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6b3b09e866a44cc5af44cdafc38ae960 --
Sep 14 10:48:15.189998 ip-10-0-136-89 systemd[1]: Starting Kubernetes Kubelet...
Sep 14 10:48:15.634285 ip-10-0-136-89 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 10:48:15.634285 ip-10-0-136-89 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 14 10:48:15.634285 ip-10-0-136-89 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 10:48:15.634285 ip-10-0-136-89 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 14 10:48:15.634285 ip-10-0-136-89 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 10:48:15.636372 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.636282    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 14 10:48:15.640483 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640465    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 10:48:15.640483 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640481    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640496    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640501    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640505    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640510    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640514    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640518    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640522    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640526    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640530    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640533    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640537    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640541    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640545    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640549    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640553    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640557    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640561    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640565    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640569    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 10:48:15.640635 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640586    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640591    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640595    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640600    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640605    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640609    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640613    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640617    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640621    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640625    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640630    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640634    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640641    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640646    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640651    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640656    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640660    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640667    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640674    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 10:48:15.641472 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640678    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640682    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640686    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640690    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640694    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640699    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640703    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640707    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640725    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640740    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640744    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640748    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640752    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640755    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640769    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640774    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640779    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640784    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640788    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640792    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 10:48:15.642256 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640796    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640801    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640805    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640809    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640813    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640818    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640822    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640827    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640831    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640835    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640840    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640844    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640848    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640852    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640856    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640861    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640867    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640871    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640875    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640880    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 10:48:15.643082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640886    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640892    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640897    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640902    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640906    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640910    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640915    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640927    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640933    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640937    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640942    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640946    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640951    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640955    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640959    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640964    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640968    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640972    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 10:48:15.643829 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.640976    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 10:48:15.644506 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644488    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 10:48:15.644506 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644504    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644509    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644512    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644516    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644519    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644523    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644525    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644528    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644531    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644535    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644539    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644542    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644545    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644548    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644550    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644553    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644556    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644558    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644561    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 10:48:15.644568 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644563    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644566    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644569    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644571    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644576    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644580    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644583    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644586    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644589    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644592    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644595    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644597    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644600    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644603    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644605    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644608    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644620    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644623    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644625    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 10:48:15.645067 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644628    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644631    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644633    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644636    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644638    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644641    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644644    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644646    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644649    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644651    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644654    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644664    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644667    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644669    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644672    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644676    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644679    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644681    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644684    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644686    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 10:48:15.645525 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644689    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644691    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644694    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644697    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644699    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644702    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644704    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644707    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644722    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644725    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644728    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644731    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644734    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644736    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644739    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644741    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644744    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644747    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644749    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644752    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 10:48:15.646032 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644754    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644757    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644760    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644762    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644771    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644773    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644776    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644780    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644782    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644785    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644788    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644790    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644793    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644795    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644798    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644802    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644805    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644808    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644811    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.644813    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 10:48:15.646520 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645225    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645230    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645234    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645237    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645240    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645243    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645245    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645248    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645251    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645253    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645256    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645258    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645261    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645263    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645266    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645268    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645276    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645279    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645283    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 10:48:15.647108 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645285    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645288    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645291    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645293    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645296    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645298    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645306    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645309    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645311    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645316    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645319    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645322    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645325    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645328    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645331    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645333    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645336    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645339    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645342    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645344    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 10:48:15.647572 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645347    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645349    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645352    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645354    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645357    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645359    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645362    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645364    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645366    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645369    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645377    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645381    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645384    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645386    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645389    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645391    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645394    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645397    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645399    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645402    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645404    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 10:48:15.648082 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645407    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645409    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645412    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645414    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645417    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645419    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645422    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645424    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645427    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645429    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645432    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645434    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645438    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645442    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645446    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645449    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645452    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645455    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645458    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 10:48:15.648647 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645461    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645464    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645466    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645475    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645478    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645480    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645483    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645485    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645488    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645491    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645493    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645496    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645498    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645501    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645503    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645505    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645508    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645510    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645513    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.645515    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 10:48:15.649137 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645586    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645617    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645627    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645632    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645635    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645639    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645643    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645647    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645650    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645654    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645657    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645661    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645664    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645667    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645670    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645673    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645682    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645685    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645692    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645695    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645698    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645701    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645704    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645708    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 14 10:48:15.649619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645724    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645727    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645730    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645734    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645737    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645739    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645743    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645746    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645750    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645753    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645756    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645759    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645762    2561 flags.go:64] FLAG: --enable-server="true"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645764    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645771    2561 flags.go:64] FLAG: --event-burst="100"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645774    2561 flags.go:64] FLAG: --event-qps="50"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645777    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645780    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645783    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645787    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645790    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645793    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645796    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645799    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645802    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 14 10:48:15.650196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645810    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645813    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645816    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645819    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645822    2561 flags.go:64] FLAG: --feature-gates=""
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645825    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645829    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645832    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645834    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645837    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645840    2561 flags.go:64] FLAG: --help="false"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645844    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-136-89.ec2.internal"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645847    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645850    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645853    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645857    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645863    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645866    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645869    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645872    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645875    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645878    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645881    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645883    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 14 10:48:15.650828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645886    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645889    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645892    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645895    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645898    2561 flags.go:64] FLAG: --lock-file=""
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645900    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645903    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645906    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645911    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645920    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645923    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645926    2561 flags.go:64] FLAG: --logging-format="text"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645928    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645932    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645935    2561 flags.go:64] FLAG: --manifest-url=""
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645937    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645941    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645944    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645952    2561 flags.go:64] FLAG: --max-pods="110"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645955    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645958    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645962    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645965    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645968    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645972    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 14 10:48:15.651415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645975    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645983    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645986    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645989    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645992    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.645995    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646001    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646003    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646006    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646009    2561 flags.go:64] FLAG: --port="10250"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646012    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646015    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f07341e8ef05a0be"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646018    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646021    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646024    2561 flags.go:64] FLAG: --register-node="true"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646027    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646031    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646039    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646042    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646045    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646049    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646052    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646055    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646057    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646060    2561 flags.go:64] FLAG: --runonce="false"
Sep 14 10:48:15.652080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646063    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646066    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646069    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646073    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646076    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646079    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646082    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646087    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646090    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646093    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646095    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646098    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646101    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646104    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646107    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646112    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646115    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646117    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646145    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646148    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646152    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646155    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646158    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646161    2561 flags.go:64] FLAG: --v="2"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646164    2561 flags.go:64] FLAG: --version="false"
Sep 14 10:48:15.652706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646175    2561 flags.go:64] FLAG: --vmodule=""
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646179    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646182    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646317    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646321    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646324    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646327    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646330    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646333    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646336    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646341    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646344    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646346    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646349    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646353    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646355    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646358    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646360    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646363    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 10:48:15.653332 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646367    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646370    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646373    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646376    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646379    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646381    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646384    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646387    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646390    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646392    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646395    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646398    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646400    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646403    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646415    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646417    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646420    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646423    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646426    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646428    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 10:48:15.653827 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646431    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646433    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646436    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646439    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646442    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646444    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646447    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646451    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646454    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646457    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646460    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646463    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646465    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646468    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646470    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646473    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646476    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646478    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646481    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646484    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 10:48:15.654671 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646486    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646489    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646491    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646494    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646496    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646500    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646504    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646513    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646516    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646519    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646522    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646525    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646527    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646530    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646533    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646537    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646539    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646542    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646545    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 10:48:15.655594 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646550    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646554    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646558    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646561    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646564    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646569    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646572    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646575    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646579    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646582    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646584    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646587    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646590    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646593    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646595    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646598    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646600    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646603    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646605    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646608    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 10:48:15.656425 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646610    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 10:48:15.657007 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646619    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 10:48:15.657007 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646622    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 10:48:15.657007 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.646625    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 10:48:15.657007 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.646630    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 10:48:15.657007 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.656493    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 14 10:48:15.657007 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.656510    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 14 10:48:15.657402 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.657390    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 14 10:48:15.657455 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.657404    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 14 10:48:15.657516 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657505    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657517    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657522    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657527    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657531    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657535    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657539    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657545    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657549    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657553    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657557    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657561    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657564    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657568    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657572    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657575    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657577    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657580    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657583    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657586    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 10:48:15.658249 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657588    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657590    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657593    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657596    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657600    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657602    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657605    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657608    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657610    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657613    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657616    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657619    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657621    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657624    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657627    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657630    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657632    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657635    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657638    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657640    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 10:48:15.658953 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657643    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657646    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657648    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657651    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657654    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657656    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657659    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657661    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657664    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657666    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657669    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657672    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657674    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657678    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657682    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657685    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657688    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657692    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657696    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657698    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 10:48:15.659644 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657701    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657703    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657706    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657722    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657727    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657729    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657733    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657735    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657738    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657741    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657743    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657746    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657748    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657751    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657754    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657757    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657759    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657762    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657764    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 10:48:15.660503 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657767    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657769    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657772    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657774    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657777    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657779    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657783    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657787    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657789    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657792    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657796    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657800    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657803    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657806    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657808    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657811    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657814    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657816    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657819    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 10:48:15.661088 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657822    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.657827    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657967    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657971    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657974    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657977    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657980    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657982    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657985    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657987    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657990    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657992    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657995    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.657997    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658000    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 10:48:15.661663 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658002    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658005    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658008    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658010    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658012    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658015    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658017    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658020    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658022    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658025    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658028    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658032    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658036    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658040    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658043    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658046    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658049    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658051    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658054    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 10:48:15.662073 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658057    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658059    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658061    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658064    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658067    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658070    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658072    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658075    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658077    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658080    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658082    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658085    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658087    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658090    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658092    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658095    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658098    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658101    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658104    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 10:48:15.662534 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658106    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658109    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658111    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658114    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658118    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658121    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658124    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658127    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658130    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658132    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658135    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658138    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658140    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658143    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658145    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658148    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658151    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658153    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658156    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 10:48:15.663060 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658158    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658161    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658163    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658166    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658169    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658171    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658174    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658176    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658179    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658181    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658194    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658198    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658203    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658207    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658211    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658215    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658219    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658223    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658227    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658232    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 10:48:15.663531 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658236    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658240    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658244    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658248    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658252    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658257    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658261    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658265    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:15.658269    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.658277    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.658417    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.661474    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.662676    2561 server.go:1019] "Starting client certificate rotation"
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.662780    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 10:48:15.664113 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.662809    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 10:48:15.667386 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.667369    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 10:48:15.673568 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.673551    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 10:48:15.684303 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.684276    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 14 10:48:15.687624 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.687608    2561 log.go:25] "Validated CRI v1 image API"
Sep 14 10:48:15.688414 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.688399    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 14 10:48:15.691772 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.691752    2561 fs.go:135] Filesystem UUIDs: map[78a0c4b4-73a5-4eca-bf44-2e508b16cb0b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cb24dada-f028-48e8-9ee0-0d4ad493dcc1:/dev/nvme0n1p4]
Sep 14 10:48:15.691840 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.691770    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 14 10:48:15.695692 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.695675    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 10:48:15.696861 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.696747    2561 manager.go:216] Machine: {Timestamp:2026-09-14 10:48:15.695401985 +0000 UTC m=+0.380086364 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3125726 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec232cf185c32c81f0c0d956f1c2973f SystemUUID:ec232cf1-85c3-2c81-f0c0-d956f1c2973f BootID:6b3b09e8-66a4-4cc5-af44-cdafc38ae960 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e7:54:62:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e7:54:62:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:28:b2:6b:32:57 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 14 10:48:15.696861 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.696857    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 14 10:48:15.696966 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.696930    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 14 10:48:15.699550 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.699530    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 14 10:48:15.699680 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.699553    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-89.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 14 10:48:15.700381 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.700371    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 14 10:48:15.700420 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.700384    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 14 10:48:15.700420 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.700392    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 10:48:15.700420 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.700405    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 10:48:15.701738 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.701728    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 14 10:48:15.706505 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.706494    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 10:48:15.706627 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.706617    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 14 10:48:15.706666 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.706645    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 14 10:48:15.706666 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.706654    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 14 10:48:15.706749 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.706667    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 14 10:48:15.706749 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.706675    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 14 10:48:15.706749 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.706683    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 14 10:48:15.707054 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.707043    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 10:48:15.707087 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.707062    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 10:48:15.710848 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.710827    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 14 10:48:15.713834 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.713819    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 14 10:48:15.713898 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.713849    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 14 10:48:15.715950 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.715939    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 14 10:48:15.715992 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.715956    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 14 10:48:15.715992 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.715963    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 14 10:48:15.715992 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.715968    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 14 10:48:15.715992 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.715977    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 14 10:48:15.715992 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.715982    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 14 10:48:15.716165 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.715996    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 14 10:48:15.716165 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.716001    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 14 10:48:15.716165 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.716007    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 14 10:48:15.716165 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.716013    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 14 10:48:15.716165 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.716021    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 14 10:48:15.716165 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.716027    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 14 10:48:15.716964 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.716952    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 14 10:48:15.717033 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.716965    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 14 10:48:15.717615 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.717596    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 14 10:48:15.717742 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.717707    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 14 10:48:15.720499 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.720487    2561 server.go:1268] "Started kubelet"
Sep 14 10:48:15.720568 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.720547    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 14 10:48:15.720707 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.720592    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 14 10:48:15.720785 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.720767    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 14 10:48:15.721286 ip-10-0-136-89 systemd[1]: Started Kubernetes Kubelet.
Sep 14 10:48:15.725189 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.725163    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 14 10:48:15.725542 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.725524    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 14 10:48:15.727130 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.727102    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 14 10:48:15.730311 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.730293    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-89.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 14 10:48:15.730822 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.730809    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 14 10:48:15.731463 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.731442    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 14 10:48:15.731559 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.730338    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-89.ec2.internal.18d52a435a93f5b4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-89.ec2.internal,UID:ip-10-0-136-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-89.ec2.internal,},FirstTimestamp:2026-09-14 10:48:15.720461748 +0000 UTC m=+0.405146128,LastTimestamp:2026-09-14 10:48:15.720461748 +0000 UTC m=+0.405146128,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-89.ec2.internal,}"
Sep 14 10:48:15.731622 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.731582    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:15.732246 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.732230    2561 factory.go:55] Registering systemd factory
Sep 14 10:48:15.732246 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.732248    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 14 10:48:15.732372 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.732315    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 14 10:48:15.732372 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.732317    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 14 10:48:15.732372 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.732342    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 14 10:48:15.732500 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.732418    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 14 10:48:15.732500 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.732428    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 14 10:48:15.732500 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.732453    2561 factory.go:153] Registering CRI-O factory
Sep 14 10:48:15.732500 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.732465    2561 factory.go:223] Registration of the crio container factory successfully
Sep 14 10:48:15.732622 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.732509    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 14 10:48:15.732622 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.732525    2561 factory.go:103] Registering Raw factory
Sep 14 10:48:15.732622 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.732534    2561 manager.go:1196] Started watching for new ooms in manager
Sep 14 10:48:15.732923 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.732911    2561 manager.go:349] Starting recovery of all containers
Sep 14 10:48:15.736109 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.736068    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 14 10:48:15.736426 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.736218    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 14 10:48:15.739460 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.739417    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 14 10:48:15.744994 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.744890    2561 manager.go:354] Recovery completed
Sep 14 10:48:15.746522 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.746397    2561 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 14 10:48:15.750490 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.750477    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:48:15.752186 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.752170    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:48:15.752246 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.752203    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:48:15.752246 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.752217    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:48:15.752767 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.752752    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 14 10:48:15.752767 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.752766    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 14 10:48:15.752901 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.752775    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:48:15.752901 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.752783    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 10:48:15.753427 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.753411    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hndfm"
Sep 14 10:48:15.753484 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.753455    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:48:15.753484 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.753479    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:48:15.753551 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.753492    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:48:15.754419 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.754346    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-89.ec2.internal.18d52a435c780ad0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-89.ec2.internal,UID:ip-10-0-136-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-89.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-89.ec2.internal,},FirstTimestamp:2026-09-14 10:48:15.752186576 +0000 UTC m=+0.436870963,LastTimestamp:2026-09-14 10:48:15.752186576 +0000 UTC m=+0.436870963,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-89.ec2.internal,}"
Sep 14 10:48:15.754988 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.754973    2561 policy_none.go:49] "None policy: Start"
Sep 14 10:48:15.754988 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.754988    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 14 10:48:15.755136 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.754997    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 14 10:48:15.756987 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.756975    2561 policy_none.go:47] "Start"
Sep 14 10:48:15.761550 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.761530    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hndfm"
Sep 14 10:48:15.764066 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.763947    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-89.ec2.internal.18d52a435c786503  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-89.ec2.internal,UID:ip-10-0-136-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-89.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-89.ec2.internal,},FirstTimestamp:2026-09-14 10:48:15.752209667 +0000 UTC m=+0.436894050,LastTimestamp:2026-09-14 10:48:15.752209667 +0000 UTC m=+0.436894050,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-89.ec2.internal,}"
Sep 14 10:48:15.768533 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.768462    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-89.ec2.internal.18d52a435c78b26f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-89.ec2.internal,UID:ip-10-0-136-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-89.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-89.ec2.internal,},FirstTimestamp:2026-09-14 10:48:15.752229487 +0000 UTC m=+0.436913867,LastTimestamp:2026-09-14 10:48:15.752229487 +0000 UTC m=+0.436913867,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-89.ec2.internal,}"
Sep 14 10:48:15.792351 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.792328    2561 manager.go:337] "Starting Device Plugin manager"
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.792368    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.792381    2561 server.go:87] "Starting device plugin registration server"
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.792673    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.792684    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.792786    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.792855    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.792864    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.793378    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.793421    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.796108    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.796164    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.796186    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.796255    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 14 10:48:15.809262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.798592    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 10:48:15.893085 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.893002    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:48:15.893980 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.893964    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:48:15.894053 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.893995    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:48:15.894053 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.894006    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:48:15.894053 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.894029    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-89.ec2.internal"
Sep 14 10:48:15.897104 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.897076    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal"]
Sep 14 10:48:15.897202 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.897140    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:48:15.898602 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.898589    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:48:15.898669 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.898615    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:48:15.898669 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.898628    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:48:15.900900 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.900888    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:48:15.901049 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.901036    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:15.901090 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.901082    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:48:15.901641 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.901625    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:48:15.901726 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.901657    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:48:15.901726 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.901671    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:48:15.901726 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.901627    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:48:15.901828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.901742    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:48:15.901828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.901753    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:48:15.902488 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.902473    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-89.ec2.internal"
Sep 14 10:48:15.902538 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.902494    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-89.ec2.internal\": node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:15.903898 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.903886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:15.903938 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.903915    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:48:15.905075 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.905062    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:48:15.905135 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.905086    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:48:15.905135 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:15.905097    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:48:15.933208 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.933192    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:15.936259 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.936244    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-89.ec2.internal\" not found" node="ip-10-0-136-89.ec2.internal"
Sep 14 10:48:15.940492 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:15.940475    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-89.ec2.internal\" not found" node="ip-10-0-136-89.ec2.internal"
Sep 14 10:48:16.033317 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:16.033287    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:16.033317 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.033312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a11e93e227e4276154b6946c45934db1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal\" (UID: \"a11e93e227e4276154b6946c45934db1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:16.033475 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.033339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a11e93e227e4276154b6946c45934db1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal\" (UID: \"a11e93e227e4276154b6946c45934db1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:16.033475 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.033355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25265a84722357db880ad05e2abf2595-config\") pod \"kube-apiserver-proxy-ip-10-0-136-89.ec2.internal\" (UID: \"25265a84722357db880ad05e2abf2595\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:16.134188 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:16.134163    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:16.134188 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.134176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a11e93e227e4276154b6946c45934db1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal\" (UID: \"a11e93e227e4276154b6946c45934db1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:16.134352 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.134200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a11e93e227e4276154b6946c45934db1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal\" (UID: \"a11e93e227e4276154b6946c45934db1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:16.134352 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.134214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25265a84722357db880ad05e2abf2595-config\") pod \"kube-apiserver-proxy-ip-10-0-136-89.ec2.internal\" (UID: \"25265a84722357db880ad05e2abf2595\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:16.134352 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.134260    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25265a84722357db880ad05e2abf2595-config\") pod \"kube-apiserver-proxy-ip-10-0-136-89.ec2.internal\" (UID: \"25265a84722357db880ad05e2abf2595\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:16.134352 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.134270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a11e93e227e4276154b6946c45934db1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal\" (UID: \"a11e93e227e4276154b6946c45934db1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:16.134352 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.134300    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a11e93e227e4276154b6946c45934db1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal\" (UID: \"a11e93e227e4276154b6946c45934db1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:16.234803 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:16.234741    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:16.239882 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.239866    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:16.245058 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.245035    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:16.335561 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:16.335536    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:16.435899 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:16.435866    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:16.536341 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:16.536274    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:16.636620 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:16.636598    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:16.662750 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.662726    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 14 10:48:16.662884 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.662865    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 14 10:48:16.736762 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:16.736736    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:16.767398 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.767365    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 10:43:15 +0000 UTC" deadline="2028-05-01 22:28:56.855450235 +0000 UTC"
Sep 14 10:48:16.767398 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.767394    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14291h40m40.088059421s"
Sep 14 10:48:16.815948 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:16.815919    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25265a84722357db880ad05e2abf2595.slice/crio-1bdc58a5ba0d61f9399692094a6b4daf4304a9162c73a0beea57dfa9665742ff WatchSource:0}: Error finding container 1bdc58a5ba0d61f9399692094a6b4daf4304a9162c73a0beea57dfa9665742ff: Status 404 returned error can't find the container with id 1bdc58a5ba0d61f9399692094a6b4daf4304a9162c73a0beea57dfa9665742ff
Sep 14 10:48:16.816357 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:16.816337    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11e93e227e4276154b6946c45934db1.slice/crio-be13ac6f10d0a5b70510c7c511ae9affa805a19ad2cb01eaf63f742aa16deef8 WatchSource:0}: Error finding container be13ac6f10d0a5b70510c7c511ae9affa805a19ad2cb01eaf63f742aa16deef8: Status 404 returned error can't find the container with id be13ac6f10d0a5b70510c7c511ae9affa805a19ad2cb01eaf63f742aa16deef8
Sep 14 10:48:16.821250 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.821229    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 10:48:16.830831 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.830812    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:48:16.831624 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.831608    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:48:16.831624 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.831635    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:48:16.831754 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.831645    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:48:16.836795 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:16.836780    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:16.907793 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.907778    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 10:48:16.937214 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:16.937197    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:16.963238 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:16.963219    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 10:48:17.037702 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:17.037681    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found"
Sep 14 10:48:17.053965 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.053946    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 10:48:17.132574 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.132512    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:17.149972 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.149953    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 10:48:17.151049 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.151037    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal"
Sep 14 10:48:17.157771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.157758    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 10:48:17.707822 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.707740    2561 apiserver.go:52] "Watching apiserver"
Sep 14 10:48:17.718987 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.718960    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 14 10:48:17.719434 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.719401    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-l5pvm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv","openshift-dns/node-resolver-nr9n4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal","openshift-multus/multus-additional-cni-plugins-qbbc4","openshift-network-diagnostics/network-check-target-5j9jp","openshift-network-operator/iptables-alerter-cl6kf","kube-system/konnectivity-agent-z6bgb","kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal","openshift-cluster-node-tuning-operator/tuned-4mmrr","openshift-image-registry/node-ca-xmbk7","openshift-multus/multus-4rjnr","openshift-multus/network-metrics-daemon-67w9c","openshift-ovn-kubernetes/ovnkube-node-8thtt"]
Sep 14 10:48:17.721997 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.721978    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.724150 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.724126    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:17.724243 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:17.724215    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:17.724566 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.724545    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 14 10:48:17.725604 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.725585    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 14 10:48:17.725693 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.725585    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 14 10:48:17.725859 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.725840    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 14 10:48:17.725961 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.725939    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-82hqr\""
Sep 14 10:48:17.726285 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.726243    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nr9n4"
Sep 14 10:48:17.728475 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.728450    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.729411 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.729389    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8hqh4\""
Sep 14 10:48:17.730244 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.729845    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 14 10:48:17.730244 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.730219    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 14 10:48:17.730798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.730780    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.731346 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.731323    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 14 10:48:17.733030 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.733011    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:17.733123 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:17.733089    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:17.733284 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.733265    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 14 10:48:17.733454 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.733437    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 14 10:48:17.733507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.733457    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lhzpd\""
Sep 14 10:48:17.734564 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.734548    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 14 10:48:17.735253 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.735238    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cl6kf"
Sep 14 10:48:17.735749 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.735732    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 14 10:48:17.735848 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.735738    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6z8b5\""
Sep 14 10:48:17.737454 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.737437    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z6bgb"
Sep 14 10:48:17.739599 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.739578    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:17.739690 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:17.739652    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:17.741754 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.741737    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6db78d45-a197-4361-9e83-fc3929395f94-cnibin\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.741849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.741761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-system-cni-dir\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.741849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.741783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-run-k8s-cni-cncf-io\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.741849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.741806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-run-netns\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.741849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.741831    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-var-lib-cni-bin\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.742039 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.741864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-etc-kubernetes\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.742039 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.741886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6db78d45-a197-4361-9e83-fc3929395f94-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.742039 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.741912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgp8\" (UniqueName: \"kubernetes.io/projected/6db78d45-a197-4361-9e83-fc3929395f94-kube-api-access-bfgp8\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.742039 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.741988    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-multus-conf-dir\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.742039 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742020    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xmbk7"
Sep 14 10:48:17.742263 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742043    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-multus-daemon-config\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.742263 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33c59f54-75fd-4277-86ad-b69ec8f49176-hosts-file\") pod \"node-resolver-nr9n4\" (UID: \"33c59f54-75fd-4277-86ad-b69ec8f49176\") " pod="openshift-dns/node-resolver-nr9n4"
Sep 14 10:48:17.742263 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg5fk\" (UniqueName: \"kubernetes.io/projected/33c59f54-75fd-4277-86ad-b69ec8f49176-kube-api-access-cg5fk\") pod \"node-resolver-nr9n4\" (UID: \"33c59f54-75fd-4277-86ad-b69ec8f49176\") " pod="openshift-dns/node-resolver-nr9n4"
Sep 14 10:48:17.742263 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742222    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-sys\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.742263 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-host\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.742496 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6db78d45-a197-4361-9e83-fc3929395f94-system-cni-dir\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.742496 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742327    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-cni-binary-copy\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.742496 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-var-lib-cni-multus\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.742496 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742416    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6db78d45-a197-4361-9e83-fc3929395f94-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.742496 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742445    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-var-lib-kubelet\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.742496 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742468    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-hostroot\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.742496 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742490    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-kubernetes\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.742496 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742492    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 14 10:48:17.742849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742547    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-tuned\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.742849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742587    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j6pm\" (UniqueName: \"kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm\") pod \"network-check-target-5j9jp\" (UID: \"659457bf-d253-49e8-becb-fddc72d02184\") " pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:17.742849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742638    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c59f54-75fd-4277-86ad-b69ec8f49176-tmp-dir\") pod \"node-resolver-nr9n4\" (UID: \"33c59f54-75fd-4277-86ad-b69ec8f49176\") " pod="openshift-dns/node-resolver-nr9n4"
Sep 14 10:48:17.742849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-modprobe-d\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.742849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742683    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-sysctl-conf\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.742849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742697    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c586d\" (UniqueName: \"kubernetes.io/projected/298e38b5-49eb-47e5-876b-71e12c70fe71-kube-api-access-c586d\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.742849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6db78d45-a197-4361-9e83-fc3929395f94-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.742849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:17.742849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742785    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-sysctl-d\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.742849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-lib-modules\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.742849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/298e38b5-49eb-47e5-876b-71e12c70fe71-tmp\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.742849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742838    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-multus-cni-dir\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.743385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-multus-socket-dir-parent\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.743385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742880    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vt4\" (UniqueName: \"kubernetes.io/projected/b3d78350-bf09-4796-9e47-d39ec74fde78-kube-api-access-q5vt4\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:17.743385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-systemd\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.743385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-run\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.743385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.742985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-var-lib-kubelet\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.743385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.743009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6db78d45-a197-4361-9e83-fc3929395f94-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.743385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.743031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-cnibin\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.743385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.743061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-run-multus-certs\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.743385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.743099    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwrqf\" (UniqueName: \"kubernetes.io/projected/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-kube-api-access-dwrqf\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.743385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.743130    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-sysconfig\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.743385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.743215    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r758b\""
Sep 14 10:48:17.743385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.743272    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 14 10:48:17.744030 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.743882    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 14 10:48:17.744030 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.744013    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 14 10:48:17.744123 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.744044    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 14 10:48:17.744123 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.744118    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-s8fkn\""
Sep 14 10:48:17.744563 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.744546    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.747030 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.747015    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.747771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.747619    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 14 10:48:17.747771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.747630    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 14 10:48:17.748884 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.748864    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 14 10:48:17.750410 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.750394    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 14 10:48:17.751037 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.751018    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 14 10:48:17.751168 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.751047    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 14 10:48:17.753534 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.752194    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 14 10:48:17.753534 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.752693    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 14 10:48:17.753534 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.753196    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 14 10:48:17.753534 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.753208    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x9vrl\""
Sep 14 10:48:17.753534 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.753265    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xrhwl\""
Sep 14 10:48:17.753831 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.753623    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qc5gr\""
Sep 14 10:48:17.753831 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.753805    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 14 10:48:17.753922 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.753859    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 14 10:48:17.754368 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.754350    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 14 10:48:17.758533 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.758514    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 10:48:17.768695 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.768675    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zkcsp"
Sep 14 10:48:17.779088 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.779070    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zkcsp"
Sep 14 10:48:17.800700 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.800668    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" event={"ID":"a11e93e227e4276154b6946c45934db1","Type":"ContainerStarted","Data":"be13ac6f10d0a5b70510c7c511ae9affa805a19ad2cb01eaf63f742aa16deef8"}
Sep 14 10:48:17.801614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.801590    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal" event={"ID":"25265a84722357db880ad05e2abf2595","Type":"ContainerStarted","Data":"1bdc58a5ba0d61f9399692094a6b4daf4304a9162c73a0beea57dfa9665742ff"}
Sep 14 10:48:17.833882 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.833864    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 14 10:48:17.843384 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c59f54-75fd-4277-86ad-b69ec8f49176-tmp-dir\") pod \"node-resolver-nr9n4\" (UID: \"33c59f54-75fd-4277-86ad-b69ec8f49176\") " pod="openshift-dns/node-resolver-nr9n4"
Sep 14 10:48:17.843497 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c586d\" (UniqueName: \"kubernetes.io/projected/298e38b5-49eb-47e5-876b-71e12c70fe71-kube-api-access-c586d\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.843497 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sht4g\" (UniqueName: \"kubernetes.io/projected/acad7b74-28e4-494a-bc19-2488bb8e2f97-kube-api-access-sht4g\") pod \"iptables-alerter-cl6kf\" (UID: \"acad7b74-28e4-494a-bc19-2488bb8e2f97\") " pod="openshift-network-operator/iptables-alerter-cl6kf"
Sep 14 10:48:17.843497 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c0aa6e02-ed99-41ac-82a1-7b9a7ab93993-agent-certs\") pod \"konnectivity-agent-z6bgb\" (UID: \"c0aa6e02-ed99-41ac-82a1-7b9a7ab93993\") " pod="kube-system/konnectivity-agent-z6bgb"
Sep 14 10:48:17.843497 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2m8v\" (UniqueName: \"kubernetes.io/projected/186e7cba-cf31-41e6-862d-29b8e1156533-kube-api-access-g2m8v\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.843497 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843496    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-run-netns\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.843750 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843521    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-run-systemd\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.843750 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843550    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:17.843750 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843574    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-sysctl-d\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.843750 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-lib-modules\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.843750 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/298e38b5-49eb-47e5-876b-71e12c70fe71-tmp\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.843750 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c0aa6e02-ed99-41ac-82a1-7b9a7ab93993-konnectivity-ca\") pod \"konnectivity-agent-z6bgb\" (UID: \"c0aa6e02-ed99-41ac-82a1-7b9a7ab93993\") " pod="kube-system/konnectivity-agent-z6bgb"
Sep 14 10:48:17.843750 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843697    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-etc-selinux\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.843750 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-run-ovn\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.844229 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:17.843755    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:17.844229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-systemd\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.844229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-run\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.844229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-var-lib-kubelet\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.844229 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:17.843834    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs podName:b3d78350-bf09-4796-9e47-d39ec74fde78 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:18.343798058 +0000 UTC m=+3.028482450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs") pod "network-metrics-daemon-67w9c" (UID: "b3d78350-bf09-4796-9e47-d39ec74fde78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:17.844229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843850    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-var-lib-kubelet\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.844229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6db78d45-a197-4361-9e83-fc3929395f94-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.844229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-run\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.844229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.843938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-systemd\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.844229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-sysctl-d\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.844229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844202    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-lib-modules\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-node-log\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844262    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-env-overrides\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwrqf\" (UniqueName: \"kubernetes.io/projected/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-kube-api-access-dwrqf\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-sysconfig\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6db78d45-a197-4361-9e83-fc3929395f94-cnibin\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acad7b74-28e4-494a-bc19-2488bb8e2f97-host-slash\") pod \"iptables-alerter-cl6kf\" (UID: \"acad7b74-28e4-494a-bc19-2488bb8e2f97\") " pod="openshift-network-operator/iptables-alerter-cl6kf"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-socket-dir\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-run-netns\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844471    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-var-lib-cni-bin\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6db78d45-a197-4361-9e83-fc3929395f94-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgp8\" (UniqueName: \"kubernetes.io/projected/6db78d45-a197-4361-9e83-fc3929395f94-kube-api-access-bfgp8\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844560    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-ovnkube-config\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-run-netns\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844642    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-sysconfig\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-multus-conf-dir\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844492    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 14 10:48:17.844771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6db78d45-a197-4361-9e83-fc3929395f94-cnibin\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844752    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6db78d45-a197-4361-9e83-fc3929395f94-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844444    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c59f54-75fd-4277-86ad-b69ec8f49176-tmp-dir\") pod \"node-resolver-nr9n4\" (UID: \"33c59f54-75fd-4277-86ad-b69ec8f49176\") " pod="openshift-dns/node-resolver-nr9n4"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844865    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-var-lib-cni-bin\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg5fk\" (UniqueName: \"kubernetes.io/projected/33c59f54-75fd-4277-86ad-b69ec8f49176-kube-api-access-cg5fk\") pod \"node-resolver-nr9n4\" (UID: \"33c59f54-75fd-4277-86ad-b69ec8f49176\") " pod="openshift-dns/node-resolver-nr9n4"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844980    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-multus-conf-dir\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.844981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-host\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6db78d45-a197-4361-9e83-fc3929395f94-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6db78d45-a197-4361-9e83-fc3929395f94-system-cni-dir\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-host\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6db78d45-a197-4361-9e83-fc3929395f94-system-cni-dir\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-dbus\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845105    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845130    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-registration-dir\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845157    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-var-lib-cni-multus\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-kubelet-config\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-var-lib-cni-multus\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.845507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-cni-bin\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-cni-netd\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845299    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-kubernetes\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-tuned\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/914ecb92-cd00-4b96-bed0-d0df7d566386-serviceca\") pod \"node-ca-xmbk7\" (UID: \"914ecb92-cd00-4b96-bed0-d0df7d566386\") " pod="openshift-image-registry/node-ca-xmbk7"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-modprobe-d\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845399    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-sysctl-conf\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-kubernetes\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845423    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6db78d45-a197-4361-9e83-fc3929395f94-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-device-dir\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-slash\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-run-openvswitch\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-modprobe-d\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-run-ovn-kubernetes\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845521    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-sysctl-conf\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845549    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-multus-cni-dir\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.846270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845572    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-multus-socket-dir-parent\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-multus-cni-dir\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vt4\" (UniqueName: \"kubernetes.io/projected/b3d78350-bf09-4796-9e47-d39ec74fde78-kube-api-access-q5vt4\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-var-lib-openvswitch\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dsx\" (UniqueName: \"kubernetes.io/projected/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-kube-api-access-w6dsx\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845653    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-multus-socket-dir-parent\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-cnibin\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-run-multus-certs\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845754    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845773    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/914ecb92-cd00-4b96-bed0-d0df7d566386-host\") pod \"node-ca-xmbk7\" (UID: \"914ecb92-cd00-4b96-bed0-d0df7d566386\") " pod="openshift-image-registry/node-ca-xmbk7"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-ovn-node-metrics-cert\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-system-cni-dir\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845852    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-run-k8s-cni-cncf-io\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845878    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-etc-kubernetes\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acad7b74-28e4-494a-bc19-2488bb8e2f97-iptables-alerter-script\") pod \"iptables-alerter-cl6kf\" (UID: \"acad7b74-28e4-494a-bc19-2488bb8e2f97\") " pod="openshift-network-operator/iptables-alerter-cl6kf"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6db78d45-a197-4361-9e83-fc3929395f94-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845934    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-multus-daemon-config\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847008 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.845973    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33c59f54-75fd-4277-86ad-b69ec8f49176-hosts-file\") pod \"node-resolver-nr9n4\" (UID: \"33c59f54-75fd-4277-86ad-b69ec8f49176\") " pod="openshift-dns/node-resolver-nr9n4"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-sys\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846094    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33c59f54-75fd-4277-86ad-b69ec8f49176-hosts-file\") pod \"node-resolver-nr9n4\" (UID: \"33c59f54-75fd-4277-86ad-b69ec8f49176\") " pod="openshift-dns/node-resolver-nr9n4"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846101    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6db78d45-a197-4361-9e83-fc3929395f94-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-cnibin\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846146    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-run-k8s-cni-cncf-io\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846150    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-kubelet\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/298e38b5-49eb-47e5-876b-71e12c70fe71-sys\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-system-cni-dir\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-etc-kubernetes\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846271    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-log-socket\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-run-multus-certs\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846386    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-cni-binary-copy\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-sys-fs\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-ovnkube-script-lib\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846486    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-var-lib-kubelet\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846509    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-hostroot\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6pm\" (UniqueName: \"kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm\") pod \"network-check-target-5j9jp\" (UID: \"659457bf-d253-49e8-becb-fddc72d02184\") " pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:17.847798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-host-var-lib-kubelet\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.848682 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846559    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xtg4\" (UniqueName: \"kubernetes.io/projected/914ecb92-cd00-4b96-bed0-d0df7d566386-kube-api-access-8xtg4\") pod \"node-ca-xmbk7\" (UID: \"914ecb92-cd00-4b96-bed0-d0df7d566386\") " pod="openshift-image-registry/node-ca-xmbk7"
Sep 14 10:48:17.848682 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-hostroot\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.848682 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-systemd-units\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.848682 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-etc-openvswitch\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.848682 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846666    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6db78d45-a197-4361-9e83-fc3929395f94-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.848682 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.846961    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-multus-daemon-config\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.848682 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.847304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-cni-binary-copy\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.848682 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.848570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/298e38b5-49eb-47e5-876b-71e12c70fe71-tmp\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.849181 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.848923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/298e38b5-49eb-47e5-876b-71e12c70fe71-etc-tuned\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.861122 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:17.861083    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:48:17.861122 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:17.861108    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:48:17.861122 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:17.861123    2561 projected.go:196] Error preparing data for projected volume kube-api-access-9j6pm for pod openshift-network-diagnostics/network-check-target-5j9jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:17.861335 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:17.861322    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm podName:659457bf-d253-49e8-becb-fddc72d02184 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:18.361300271 +0000 UTC m=+3.045984652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9j6pm" (UniqueName: "kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm") pod "network-check-target-5j9jp" (UID: "659457bf-d253-49e8-becb-fddc72d02184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:17.863423 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.863399    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c586d\" (UniqueName: \"kubernetes.io/projected/298e38b5-49eb-47e5-876b-71e12c70fe71-kube-api-access-c586d\") pod \"tuned-4mmrr\" (UID: \"298e38b5-49eb-47e5-876b-71e12c70fe71\") " pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:17.863700 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.863682    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgp8\" (UniqueName: \"kubernetes.io/projected/6db78d45-a197-4361-9e83-fc3929395f94-kube-api-access-bfgp8\") pod \"multus-additional-cni-plugins-qbbc4\" (UID: \"6db78d45-a197-4361-9e83-fc3929395f94\") " pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:17.864318 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.864297    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vt4\" (UniqueName: \"kubernetes.io/projected/b3d78350-bf09-4796-9e47-d39ec74fde78-kube-api-access-q5vt4\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:17.864397 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.864323    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg5fk\" (UniqueName: \"kubernetes.io/projected/33c59f54-75fd-4277-86ad-b69ec8f49176-kube-api-access-cg5fk\") pod \"node-resolver-nr9n4\" (UID: \"33c59f54-75fd-4277-86ad-b69ec8f49176\") " pod="openshift-dns/node-resolver-nr9n4"
Sep 14 10:48:17.865285 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.865238    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwrqf\" (UniqueName: \"kubernetes.io/projected/20aafb29-e7c1-4ab2-94ab-a872003f7ae1-kube-api-access-dwrqf\") pod \"multus-4rjnr\" (UID: \"20aafb29-e7c1-4ab2-94ab-a872003f7ae1\") " pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:17.946860 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.946829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-dbus\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:17.946860 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.946860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.947054 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.946883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-registration-dir\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.947054 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.946936    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.947054 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.946954    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-registration-dir\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.947054 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.946972    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-dbus\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:17.947054 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.946987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-kubelet-config\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:17.947054 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947052    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-kubelet-config\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:17.947274 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947053    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-cni-bin\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947274 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-cni-netd\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947274 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947274 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947138    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-cni-bin\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947274 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947152    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/914ecb92-cd00-4b96-bed0-d0df7d566386-serviceca\") pod \"node-ca-xmbk7\" (UID: \"914ecb92-cd00-4b96-bed0-d0df7d566386\") " pod="openshift-image-registry/node-ca-xmbk7"
Sep 14 10:48:17.947274 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-cni-netd\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947274 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947173    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947274 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947177    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-device-dir\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.947274 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-slash\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947274 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-run-openvswitch\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947274 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-device-dir\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.947274 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947248    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-run-ovn-kubernetes\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947274 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-var-lib-openvswitch\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-run-openvswitch\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947288    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-run-ovn-kubernetes\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947306    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dsx\" (UniqueName: \"kubernetes.io/projected/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-kube-api-access-w6dsx\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947326    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-var-lib-openvswitch\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-slash\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/914ecb92-cd00-4b96-bed0-d0df7d566386-host\") pod \"node-ca-xmbk7\" (UID: \"914ecb92-cd00-4b96-bed0-d0df7d566386\") " pod="openshift-image-registry/node-ca-xmbk7"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-ovn-node-metrics-cert\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947411    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acad7b74-28e4-494a-bc19-2488bb8e2f97-iptables-alerter-script\") pod \"iptables-alerter-cl6kf\" (UID: \"acad7b74-28e4-494a-bc19-2488bb8e2f97\") " pod="openshift-network-operator/iptables-alerter-cl6kf"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:17.947438    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/914ecb92-cd00-4b96-bed0-d0df7d566386-host\") pod \"node-ca-xmbk7\" (UID: \"914ecb92-cd00-4b96-bed0-d0df7d566386\") " pod="openshift-image-registry/node-ca-xmbk7"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:17.947533    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret podName:07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:18.447515602 +0000 UTC m=+3.132199972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret") pod "global-pull-secret-syncer-l5pvm" (UID: "07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947440    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-kubelet\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947553    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-kubelet\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947567    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-log-socket\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947591    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/914ecb92-cd00-4b96-bed0-d0df7d566386-serviceca\") pod \"node-ca-xmbk7\" (UID: \"914ecb92-cd00-4b96-bed0-d0df7d566386\") " pod="openshift-image-registry/node-ca-xmbk7"
Sep 14 10:48:17.947902 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-sys-fs\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947632    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-log-socket\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-ovnkube-script-lib\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-sys-fs\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtg4\" (UniqueName: \"kubernetes.io/projected/914ecb92-cd00-4b96-bed0-d0df7d566386-kube-api-access-8xtg4\") pod \"node-ca-xmbk7\" (UID: \"914ecb92-cd00-4b96-bed0-d0df7d566386\") " pod="openshift-image-registry/node-ca-xmbk7"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-systemd-units\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-etc-openvswitch\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947791    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sht4g\" (UniqueName: \"kubernetes.io/projected/acad7b74-28e4-494a-bc19-2488bb8e2f97-kube-api-access-sht4g\") pod \"iptables-alerter-cl6kf\" (UID: \"acad7b74-28e4-494a-bc19-2488bb8e2f97\") " pod="openshift-network-operator/iptables-alerter-cl6kf"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c0aa6e02-ed99-41ac-82a1-7b9a7ab93993-agent-certs\") pod \"konnectivity-agent-z6bgb\" (UID: \"c0aa6e02-ed99-41ac-82a1-7b9a7ab93993\") " pod="kube-system/konnectivity-agent-z6bgb"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2m8v\" (UniqueName: \"kubernetes.io/projected/186e7cba-cf31-41e6-862d-29b8e1156533-kube-api-access-g2m8v\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947886    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-run-netns\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947906    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-systemd-units\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947916    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-run-systemd\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acad7b74-28e4-494a-bc19-2488bb8e2f97-iptables-alerter-script\") pod \"iptables-alerter-cl6kf\" (UID: \"acad7b74-28e4-494a-bc19-2488bb8e2f97\") " pod="openshift-network-operator/iptables-alerter-cl6kf"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.947962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-host-run-netns\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c0aa6e02-ed99-41ac-82a1-7b9a7ab93993-konnectivity-ca\") pod \"konnectivity-agent-z6bgb\" (UID: \"c0aa6e02-ed99-41ac-82a1-7b9a7ab93993\") " pod="kube-system/konnectivity-agent-z6bgb"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-etc-selinux\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.948614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948074    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-run-systemd\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948118    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-etc-openvswitch\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948135    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-ovnkube-script-lib\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-run-ovn\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-node-log\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948193    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-env-overrides\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acad7b74-28e4-494a-bc19-2488bb8e2f97-host-slash\") pod \"iptables-alerter-cl6kf\" (UID: \"acad7b74-28e4-494a-bc19-2488bb8e2f97\") " pod="openshift-network-operator/iptables-alerter-cl6kf"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-run-ovn\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-socket-dir\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948275    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-node-log\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-ovnkube-config\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-socket-dir\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/186e7cba-cf31-41e6-862d-29b8e1156533-etc-selinux\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948547    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acad7b74-28e4-494a-bc19-2488bb8e2f97-host-slash\") pod \"iptables-alerter-cl6kf\" (UID: \"acad7b74-28e4-494a-bc19-2488bb8e2f97\") " pod="openshift-network-operator/iptables-alerter-cl6kf"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948552    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c0aa6e02-ed99-41ac-82a1-7b9a7ab93993-konnectivity-ca\") pod \"konnectivity-agent-z6bgb\" (UID: \"c0aa6e02-ed99-41ac-82a1-7b9a7ab93993\") " pod="kube-system/konnectivity-agent-z6bgb"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-env-overrides\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.949277 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.948821    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-ovnkube-config\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.950275 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.950250    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-ovn-node-metrics-cert\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.950366 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.950334    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c0aa6e02-ed99-41ac-82a1-7b9a7ab93993-agent-certs\") pod \"konnectivity-agent-z6bgb\" (UID: \"c0aa6e02-ed99-41ac-82a1-7b9a7ab93993\") " pod="kube-system/konnectivity-agent-z6bgb"
Sep 14 10:48:17.958094 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.958045    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dsx\" (UniqueName: \"kubernetes.io/projected/7bb39c20-00eb-45b3-9fd7-deb3c2040e16-kube-api-access-w6dsx\") pod \"ovnkube-node-8thtt\" (UID: \"7bb39c20-00eb-45b3-9fd7-deb3c2040e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:17.963769 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.963704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2m8v\" (UniqueName: \"kubernetes.io/projected/186e7cba-cf31-41e6-862d-29b8e1156533-kube-api-access-g2m8v\") pod \"aws-ebs-csi-driver-node-pf5vv\" (UID: \"186e7cba-cf31-41e6-862d-29b8e1156533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:17.964184 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.964162    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtg4\" (UniqueName: \"kubernetes.io/projected/914ecb92-cd00-4b96-bed0-d0df7d566386-kube-api-access-8xtg4\") pod \"node-ca-xmbk7\" (UID: \"914ecb92-cd00-4b96-bed0-d0df7d566386\") " pod="openshift-image-registry/node-ca-xmbk7"
Sep 14 10:48:17.964479 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:17.964457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sht4g\" (UniqueName: \"kubernetes.io/projected/acad7b74-28e4-494a-bc19-2488bb8e2f97-kube-api-access-sht4g\") pod \"iptables-alerter-cl6kf\" (UID: \"acad7b74-28e4-494a-bc19-2488bb8e2f97\") " pod="openshift-network-operator/iptables-alerter-cl6kf"
Sep 14 10:48:18.018042 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.018009    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 10:48:18.033592 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.033564    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4rjnr"
Sep 14 10:48:18.039829 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.039803    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nr9n4"
Sep 14 10:48:18.048432 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.048412    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4mmrr"
Sep 14 10:48:18.057853 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.057829    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qbbc4"
Sep 14 10:48:18.064457 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.064439    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cl6kf"
Sep 14 10:48:18.072056 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.072034    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z6bgb"
Sep 14 10:48:18.077772 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.077755    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xmbk7"
Sep 14 10:48:18.084477 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.084454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv"
Sep 14 10:48:18.090225 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.090207    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:18.135021 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:18.134869    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c59f54_75fd_4277_86ad_b69ec8f49176.slice/crio-73f6187c359406c5a1fbf20a982613a5e80bfccd6d0b3efb98e4393316e0ab18 WatchSource:0}: Error finding container 73f6187c359406c5a1fbf20a982613a5e80bfccd6d0b3efb98e4393316e0ab18: Status 404 returned error can't find the container with id 73f6187c359406c5a1fbf20a982613a5e80bfccd6d0b3efb98e4393316e0ab18
Sep 14 10:48:18.135612 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:18.135582    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb39c20_00eb_45b3_9fd7_deb3c2040e16.slice/crio-a2ecd7eda0c7fd25fc62c7516ed21beb614b7e0efaeeaedc760ae2fbbdc23b82 WatchSource:0}: Error finding container a2ecd7eda0c7fd25fc62c7516ed21beb614b7e0efaeeaedc760ae2fbbdc23b82: Status 404 returned error can't find the container with id a2ecd7eda0c7fd25fc62c7516ed21beb614b7e0efaeeaedc760ae2fbbdc23b82
Sep 14 10:48:18.137371 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:18.137350    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914ecb92_cd00_4b96_bed0_d0df7d566386.slice/crio-5f456f77671009be90ca78c8102dba3f4b9752fbd88adf66972e86c2416eecc8 WatchSource:0}: Error finding container 5f456f77671009be90ca78c8102dba3f4b9752fbd88adf66972e86c2416eecc8: Status 404 returned error can't find the container with id 5f456f77671009be90ca78c8102dba3f4b9752fbd88adf66972e86c2416eecc8
Sep 14 10:48:18.137973 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:18.137891    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db78d45_a197_4361_9e83_fc3929395f94.slice/crio-81356cef3c50e57afe8aa53b810df8882deef0f3222d488e80b92b7ce9f20866 WatchSource:0}: Error finding container 81356cef3c50e57afe8aa53b810df8882deef0f3222d488e80b92b7ce9f20866: Status 404 returned error can't find the container with id 81356cef3c50e57afe8aa53b810df8882deef0f3222d488e80b92b7ce9f20866
Sep 14 10:48:18.139156 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:18.139131    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298e38b5_49eb_47e5_876b_71e12c70fe71.slice/crio-831d75f5982c17a510ac0f50ded5fdd4508e4a828fbc91e6cfc0363f51feb13b WatchSource:0}: Error finding container 831d75f5982c17a510ac0f50ded5fdd4508e4a828fbc91e6cfc0363f51feb13b: Status 404 returned error can't find the container with id 831d75f5982c17a510ac0f50ded5fdd4508e4a828fbc91e6cfc0363f51feb13b
Sep 14 10:48:18.141063 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:18.141040    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0aa6e02_ed99_41ac_82a1_7b9a7ab93993.slice/crio-5cc9fa96ce9c2a2b05c158cc63193c673baa9c34c1dd24df120550ffc3a3b436 WatchSource:0}: Error finding container 5cc9fa96ce9c2a2b05c158cc63193c673baa9c34c1dd24df120550ffc3a3b436: Status 404 returned error can't find the container with id 5cc9fa96ce9c2a2b05c158cc63193c673baa9c34c1dd24df120550ffc3a3b436
Sep 14 10:48:18.351469 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.351397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:18.351586 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:18.351502    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:18.351586 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:18.351549    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs podName:b3d78350-bf09-4796-9e47-d39ec74fde78 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:19.351536745 +0000 UTC m=+4.036221117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs") pod "network-metrics-daemon-67w9c" (UID: "b3d78350-bf09-4796-9e47-d39ec74fde78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:18.451978 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.451956    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6pm\" (UniqueName: \"kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm\") pod \"network-check-target-5j9jp\" (UID: \"659457bf-d253-49e8-becb-fddc72d02184\") " pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:18.452071 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.452015    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:18.452134 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:18.452115    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:18.452181 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:18.452172    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret podName:07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:19.45215877 +0000 UTC m=+4.136843141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret") pod "global-pull-secret-syncer-l5pvm" (UID: "07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:18.452225 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:18.452125    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:48:18.452225 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:18.452219    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:48:18.452294 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:18.452231    2561 projected.go:196] Error preparing data for projected volume kube-api-access-9j6pm for pod openshift-network-diagnostics/network-check-target-5j9jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:18.452294 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:18.452286    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm podName:659457bf-d253-49e8-becb-fddc72d02184 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:19.452274293 +0000 UTC m=+4.136958673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9j6pm" (UniqueName: "kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm") pod "network-check-target-5j9jp" (UID: "659457bf-d253-49e8-becb-fddc72d02184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:18.783038 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.782941    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 10:43:17 +0000 UTC" deadline="2028-02-08 22:11:35.44251819 +0000 UTC"
Sep 14 10:48:18.783038 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.782993    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12299h23m16.659529768s"
Sep 14 10:48:18.817206 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.817170    2561 generic.go:356] "Generic (PLEG): container finished" podID="a11e93e227e4276154b6946c45934db1" containerID="f250d1d1895c3fa331ebda797894a1f3af6b72d4ce1cfd10969e06da43abeb86" exitCode=0
Sep 14 10:48:18.817349 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.817258    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" event={"ID":"a11e93e227e4276154b6946c45934db1","Type":"ContainerDied","Data":"f250d1d1895c3fa331ebda797894a1f3af6b72d4ce1cfd10969e06da43abeb86"}
Sep 14 10:48:18.825758 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.825727    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4rjnr" event={"ID":"20aafb29-e7c1-4ab2-94ab-a872003f7ae1","Type":"ContainerStarted","Data":"25722fa02312c560e2b38b26d551ed906a517d8ed0505ebd1d4bd6878ed1017d"}
Sep 14 10:48:18.830057 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.830029    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z6bgb" event={"ID":"c0aa6e02-ed99-41ac-82a1-7b9a7ab93993","Type":"ContainerStarted","Data":"5cc9fa96ce9c2a2b05c158cc63193c673baa9c34c1dd24df120550ffc3a3b436"}
Sep 14 10:48:18.834610 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.834452    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbbc4" event={"ID":"6db78d45-a197-4361-9e83-fc3929395f94","Type":"ContainerStarted","Data":"81356cef3c50e57afe8aa53b810df8882deef0f3222d488e80b92b7ce9f20866"}
Sep 14 10:48:18.839837 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.839740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt" event={"ID":"7bb39c20-00eb-45b3-9fd7-deb3c2040e16","Type":"ContainerStarted","Data":"a2ecd7eda0c7fd25fc62c7516ed21beb614b7e0efaeeaedc760ae2fbbdc23b82"}
Sep 14 10:48:18.845840 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.845814    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nr9n4" event={"ID":"33c59f54-75fd-4277-86ad-b69ec8f49176","Type":"ContainerStarted","Data":"73f6187c359406c5a1fbf20a982613a5e80bfccd6d0b3efb98e4393316e0ab18"}
Sep 14 10:48:18.862400 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.862291    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv" event={"ID":"186e7cba-cf31-41e6-862d-29b8e1156533","Type":"ContainerStarted","Data":"55313b4dfae63742e243b6f1b4f893be4c99d0aa2efd11b880b0c831f6a5459c"}
Sep 14 10:48:18.871809 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.871737    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cl6kf" event={"ID":"acad7b74-28e4-494a-bc19-2488bb8e2f97","Type":"ContainerStarted","Data":"4b016401965919fb6b90cde3a8fcfa4ace28e8b1bf435cbe62aaafc8b1d3b24a"}
Sep 14 10:48:18.877222 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.877194    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4mmrr" event={"ID":"298e38b5-49eb-47e5-876b-71e12c70fe71","Type":"ContainerStarted","Data":"831d75f5982c17a510ac0f50ded5fdd4508e4a828fbc91e6cfc0363f51feb13b"}
Sep 14 10:48:18.884279 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:18.884209    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xmbk7" event={"ID":"914ecb92-cd00-4b96-bed0-d0df7d566386","Type":"ContainerStarted","Data":"5f456f77671009be90ca78c8102dba3f4b9752fbd88adf66972e86c2416eecc8"}
Sep 14 10:48:19.364385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:19.364305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:19.364538 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:19.364508    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:19.364607 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:19.364568    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs podName:b3d78350-bf09-4796-9e47-d39ec74fde78 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:21.364551029 +0000 UTC m=+6.049235412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs") pod "network-metrics-daemon-67w9c" (UID: "b3d78350-bf09-4796-9e47-d39ec74fde78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:19.464910 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:19.464872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:19.465080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:19.464937    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6pm\" (UniqueName: \"kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm\") pod \"network-check-target-5j9jp\" (UID: \"659457bf-d253-49e8-becb-fddc72d02184\") " pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:19.465138 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:19.465121    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:48:19.465187 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:19.465142    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:48:19.465187 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:19.465155    2561 projected.go:196] Error preparing data for projected volume kube-api-access-9j6pm for pod openshift-network-diagnostics/network-check-target-5j9jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:19.465305 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:19.465207    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm podName:659457bf-d253-49e8-becb-fddc72d02184 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:21.465190249 +0000 UTC m=+6.149874616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9j6pm" (UniqueName: "kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm") pod "network-check-target-5j9jp" (UID: "659457bf-d253-49e8-becb-fddc72d02184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:19.465593 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:19.465577    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:19.465669 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:19.465629    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret podName:07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:21.465613637 +0000 UTC m=+6.150298018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret") pod "global-pull-secret-syncer-l5pvm" (UID: "07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:19.783382 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:19.783269    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 10:43:17 +0000 UTC" deadline="2028-03-09 10:45:44.023925786 +0000 UTC"
Sep 14 10:48:19.783382 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:19.783329    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13007h57m24.240600995s"
Sep 14 10:48:19.796859 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:19.796835    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:19.796990 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:19.796971    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:19.797514 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:19.797362    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:19.797514 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:19.797454    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:19.797640 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:19.797519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:19.797640 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:19.797589    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:19.914766 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:19.914128    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" event={"ID":"a11e93e227e4276154b6946c45934db1","Type":"ContainerStarted","Data":"0f66793efa709f870498fef1901bab5c9d36499dc1c4730f349a7a471c16563b"}
Sep 14 10:48:21.384024 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:21.383989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:21.384608 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:21.384126    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:21.384608 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:21.384174    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs podName:b3d78350-bf09-4796-9e47-d39ec74fde78 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:25.384161951 +0000 UTC m=+10.068846317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs") pod "network-metrics-daemon-67w9c" (UID: "b3d78350-bf09-4796-9e47-d39ec74fde78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:21.485205 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:21.485160    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:21.485377 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:21.485218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6pm\" (UniqueName: \"kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm\") pod \"network-check-target-5j9jp\" (UID: \"659457bf-d253-49e8-becb-fddc72d02184\") " pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:21.485377 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:21.485367    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:48:21.485483 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:21.485385    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:48:21.485483 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:21.485397    2561 projected.go:196] Error preparing data for projected volume kube-api-access-9j6pm for pod openshift-network-diagnostics/network-check-target-5j9jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:21.485483 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:21.485453    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm podName:659457bf-d253-49e8-becb-fddc72d02184 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:25.485432926 +0000 UTC m=+10.170117300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9j6pm" (UniqueName: "kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm") pod "network-check-target-5j9jp" (UID: "659457bf-d253-49e8-becb-fddc72d02184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:21.486002 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:21.485977    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:21.486114 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:21.486041    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret podName:07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:25.486023499 +0000 UTC m=+10.170707866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret") pod "global-pull-secret-syncer-l5pvm" (UID: "07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:21.797353 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:21.797282    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:21.797496 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:21.797417    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:21.797784 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:21.797760    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:21.797891 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:21.797865    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:21.797948 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:21.797911    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:21.798006 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:21.797981    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:23.797077 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:23.796980    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:23.797481 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:23.796980    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:23.797481 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:23.797136    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:23.797481 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:23.797150    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:23.797481 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:23.797326    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:23.797481 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:23.797412    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:25.423305 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:25.423273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:25.423674 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:25.423409    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:25.423674 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:25.423458    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs podName:b3d78350-bf09-4796-9e47-d39ec74fde78 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:33.423445542 +0000 UTC m=+18.108129914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs") pod "network-metrics-daemon-67w9c" (UID: "b3d78350-bf09-4796-9e47-d39ec74fde78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:25.524446 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:25.523609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:25.524446 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:25.523667    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6pm\" (UniqueName: \"kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm\") pod \"network-check-target-5j9jp\" (UID: \"659457bf-d253-49e8-becb-fddc72d02184\") " pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:25.524446 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:25.523861    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:48:25.524446 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:25.523882    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:48:25.524446 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:25.523901    2561 projected.go:196] Error preparing data for projected volume kube-api-access-9j6pm for pod openshift-network-diagnostics/network-check-target-5j9jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:25.524446 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:25.523963    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm podName:659457bf-d253-49e8-becb-fddc72d02184 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:33.523944256 +0000 UTC m=+18.208628630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9j6pm" (UniqueName: "kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm") pod "network-check-target-5j9jp" (UID: "659457bf-d253-49e8-becb-fddc72d02184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:25.524446 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:25.524360    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:25.524446 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:25.524410    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret podName:07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:33.524394432 +0000 UTC m=+18.209078806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret") pod "global-pull-secret-syncer-l5pvm" (UID: "07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:25.797934 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:25.797852    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:25.798096 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:25.797952    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:25.798096 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:25.797982    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:25.798096 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:25.797988    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:25.798249 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:25.798091    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:25.798249 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:25.798166    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:27.797445 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:27.797413    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:27.797921 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:27.797413    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:27.797921 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:27.797529    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:27.797921 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:27.797541    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:27.797921 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:27.797625    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:27.797921 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:27.797742    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:29.797404 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:29.797368    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:29.797892 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:29.797368    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:29.797892 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:29.797501    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:29.797892 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:29.797574    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:29.797892 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:29.797372    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:29.797892 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:29.797680    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:31.797518 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:31.797216    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:31.797518 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:31.797233    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:31.797518 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:31.797364    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:31.797518 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:31.797419    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:31.798075 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:31.797530    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:31.798075 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:31.797614    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:33.483593 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:33.483506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:33.484013 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:33.483662    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:33.484013 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:33.483731    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs podName:b3d78350-bf09-4796-9e47-d39ec74fde78 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:49.483703666 +0000 UTC m=+34.168388033 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs") pod "network-metrics-daemon-67w9c" (UID: "b3d78350-bf09-4796-9e47-d39ec74fde78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:33.584027 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:33.583988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:33.584027 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:33.584034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6pm\" (UniqueName: \"kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm\") pod \"network-check-target-5j9jp\" (UID: \"659457bf-d253-49e8-becb-fddc72d02184\") " pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:33.584242 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:33.584146    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:33.584242 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:33.584169    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:48:33.584242 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:33.584186    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:48:33.584242 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:33.584199    2561 projected.go:196] Error preparing data for projected volume kube-api-access-9j6pm for pod openshift-network-diagnostics/network-check-target-5j9jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:33.584242 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:33.584212    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret podName:07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:49.584192989 +0000 UTC m=+34.268877375 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret") pod "global-pull-secret-syncer-l5pvm" (UID: "07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:33.584242 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:33.584244    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm podName:659457bf-d253-49e8-becb-fddc72d02184 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:49.58423015 +0000 UTC m=+34.268914531 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9j6pm" (UniqueName: "kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm") pod "network-check-target-5j9jp" (UID: "659457bf-d253-49e8-becb-fddc72d02184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:33.797451 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:33.797371    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:33.797451 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:33.797396    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:33.797663 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:33.797371    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:33.797663 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:33.797487    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:33.797663 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:33.797612    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:33.797825 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:33.797696    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:35.797696 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:35.797470    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:35.798142 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:35.797567    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:35.798142 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:35.797591    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:35.798142 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:35.797799    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:35.798142 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:35.797876    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:35.798142 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:35.797966    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:36.967025 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:36.966988    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal" event={"ID":"25265a84722357db880ad05e2abf2595","Type":"ContainerStarted","Data":"224cc79753945f112d6f67a01e8aec025343083581d3c8cbc3b4b49db2657759"}
Sep 14 10:48:36.982329 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:36.982270    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal" podStartSLOduration=19.982232663 podStartE2EDuration="19.982232663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:36.982136648 +0000 UTC m=+21.666821039" watchObservedRunningTime="2026-09-14 10:48:36.982232663 +0000 UTC m=+21.666917051"
Sep 14 10:48:36.983151 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:36.983113    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" podStartSLOduration=19.983094998 podStartE2EDuration="19.983094998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:19.933488045 +0000 UTC m=+4.618172437" watchObservedRunningTime="2026-09-14 10:48:36.983094998 +0000 UTC m=+21.667779391"
Sep 14 10:48:37.796927 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.796685    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:37.797092 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.796693    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:37.797092 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:37.797038    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:37.797092 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.796705    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:37.797236 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:37.797114    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:37.797236 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:37.797146    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:37.971117 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.971085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4rjnr" event={"ID":"20aafb29-e7c1-4ab2-94ab-a872003f7ae1","Type":"ContainerStarted","Data":"a01cf366405904d6db5d77a8bb53a61f7e8d2dab62c959f99439767379ab34e9"}
Sep 14 10:48:37.972733 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.972698    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z6bgb" event={"ID":"c0aa6e02-ed99-41ac-82a1-7b9a7ab93993","Type":"ContainerStarted","Data":"f7fdfc44be07f1b3c39304b274bbf9a5d56ab4d7a4f4c1fb0e0dfc4f8d18faae"}
Sep 14 10:48:37.974312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.974286    2561 generic.go:356] "Generic (PLEG): container finished" podID="6db78d45-a197-4361-9e83-fc3929395f94" containerID="7c271efff2b31225db3fbe098c4baedfaf6d45a5eaa2924186e4a30d459f322a" exitCode=0
Sep 14 10:48:37.974395 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.974361    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbbc4" event={"ID":"6db78d45-a197-4361-9e83-fc3929395f94","Type":"ContainerDied","Data":"7c271efff2b31225db3fbe098c4baedfaf6d45a5eaa2924186e4a30d459f322a"}
Sep 14 10:48:37.977852 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.977829    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt" event={"ID":"7bb39c20-00eb-45b3-9fd7-deb3c2040e16","Type":"ContainerStarted","Data":"d241166e608d4facc1d8f92af1cdb610cc5d238253b2034ec8b66ccb55219f2d"}
Sep 14 10:48:37.977935 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.977888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt" event={"ID":"7bb39c20-00eb-45b3-9fd7-deb3c2040e16","Type":"ContainerStarted","Data":"34318768f3260c4587e90c706df467f75e4000a031497949019c57d2eb12ca79"}
Sep 14 10:48:37.977935 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.977929    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt" event={"ID":"7bb39c20-00eb-45b3-9fd7-deb3c2040e16","Type":"ContainerStarted","Data":"e11c156c4b46fa87db33f1e4dc6f6f8ca2a32ca0bcd4e860b83530393cd5fd5d"}
Sep 14 10:48:37.978021 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.977977    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt" event={"ID":"7bb39c20-00eb-45b3-9fd7-deb3c2040e16","Type":"ContainerStarted","Data":"15c12db1d83bf5f780e515672cf8bf984043e1d872aa414c56b1c1651eae4ce6"}
Sep 14 10:48:37.978065 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.978034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt" event={"ID":"7bb39c20-00eb-45b3-9fd7-deb3c2040e16","Type":"ContainerStarted","Data":"1a27bc123455b634a92d3ff4da694453d49bbaeabb9f1e0e231f04b01545213d"}
Sep 14 10:48:37.978108 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.978085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt" event={"ID":"7bb39c20-00eb-45b3-9fd7-deb3c2040e16","Type":"ContainerStarted","Data":"234320c04e19e35fe9f1ebaecf087fb7edc62ea34cf26da55005c3da5a5e1796"}
Sep 14 10:48:37.979229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.979208    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nr9n4" event={"ID":"33c59f54-75fd-4277-86ad-b69ec8f49176","Type":"ContainerStarted","Data":"b2386a02ca4316677d259ad46c7da62138bd53f3d1b7e5a0811284d50649eafa"}
Sep 14 10:48:37.980933 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.980913    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv" event={"ID":"186e7cba-cf31-41e6-862d-29b8e1156533","Type":"ContainerStarted","Data":"54f6af52570312ec6a7a04b87dd686e0cadb3540d6437db6632498486d710e47"}
Sep 14 10:48:37.982563 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.982532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4mmrr" event={"ID":"298e38b5-49eb-47e5-876b-71e12c70fe71","Type":"ContainerStarted","Data":"32b7d7b6e4c4988b24bc2452ae7a9fc846065b8bd7e5447217487ebc2341a638"}
Sep 14 10:48:37.984015 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.983993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xmbk7" event={"ID":"914ecb92-cd00-4b96-bed0-d0df7d566386","Type":"ContainerStarted","Data":"f5fe9c89dea535319e1f367efbc488569dbe1b0d8af336c3c5692d6661762cea"}
Sep 14 10:48:37.992869 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:37.992833    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4rjnr" podStartSLOduration=4.249617882 podStartE2EDuration="22.992820775s" totalImagesPullingTime="18.743202893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:37.992179568 +0000 UTC m=+22.676863958" watchObservedRunningTime="2026-09-14 10:48:37.992820775 +0000 UTC m=+22.677505164"
Sep 14 10:48:38.027383 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:38.027342    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4mmrr" podStartSLOduration=3.312020566 podStartE2EDuration="22.02732823s" totalImagesPullingTime="18.715307664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:38.026911769 +0000 UTC m=+22.711596194" watchObservedRunningTime="2026-09-14 10:48:38.02732823 +0000 UTC m=+22.712012618"
Sep 14 10:48:38.043065 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:38.043026    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z6bgb" podStartSLOduration=3.363478396 podStartE2EDuration="22.043012977s" totalImagesPullingTime="18.679534581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:38.042869589 +0000 UTC m=+22.727553978" watchObservedRunningTime="2026-09-14 10:48:38.043012977 +0000 UTC m=+22.727697386"
Sep 14 10:48:38.069207 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:38.069165    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xmbk7" podStartSLOduration=3.386391342 podStartE2EDuration="22.069152554s" totalImagesPullingTime="18.682761212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:38.066838326 +0000 UTC m=+22.751522714" watchObservedRunningTime="2026-09-14 10:48:38.069152554 +0000 UTC m=+22.753836942"
Sep 14 10:48:38.089671 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:38.089624    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nr9n4" podStartSLOduration=4.403651743 podStartE2EDuration="23.089608645s" totalImagesPullingTime="18.685956902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:38.088809693 +0000 UTC m=+22.773494082" watchObservedRunningTime="2026-09-14 10:48:38.089608645 +0000 UTC m=+22.774293034"
Sep 14 10:48:38.557537 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:38.557363    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 14 10:48:38.809528 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:38.809396    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T10:48:38.557533772Z","UUID":"1c3abaac-7c4e-4569-b659-90fce1d9ef8c","Handler":null,"Name":"","Endpoint":""}
Sep 14 10:48:38.810771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:38.810753    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 14 10:48:38.810888 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:38.810779    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 14 10:48:38.988196 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:38.988164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv" event={"ID":"186e7cba-cf31-41e6-862d-29b8e1156533","Type":"ContainerStarted","Data":"569a85c76b7204f7a25a4a5ea8bfa266d541c2d408c4115329db7994c985baf2"}
Sep 14 10:48:38.989825 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:38.989789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cl6kf" event={"ID":"acad7b74-28e4-494a-bc19-2488bb8e2f97","Type":"ContainerStarted","Data":"fbfbe563738b15ed4ce5ad52ec53253af31e3dcebc3dd5d6019065571d89d451"}
Sep 14 10:48:39.008536 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:39.007082    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cl6kf" podStartSLOduration=4.346127167 podStartE2EDuration="23.007062996s" totalImagesPullingTime="18.660935829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:39.005553807 +0000 UTC m=+23.690238197" watchObservedRunningTime="2026-09-14 10:48:39.007062996 +0000 UTC m=+23.691747386"
Sep 14 10:48:39.709246 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:39.709219    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z6bgb"
Sep 14 10:48:39.709820 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:39.709794    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z6bgb"
Sep 14 10:48:39.796945 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:39.796882    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:39.797080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:39.796883    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:39.797080 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:39.796993    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:39.797211 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:39.797129    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:39.797211 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:39.797193    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:39.797320 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:39.797294    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:39.996057 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:39.996029    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt" event={"ID":"7bb39c20-00eb-45b3-9fd7-deb3c2040e16","Type":"ContainerStarted","Data":"1e1c2d5cbff89521f78ec50d096f5ab130d76500b669c562ba3a91f86540d034"}
Sep 14 10:48:39.996460 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:39.996395    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z6bgb"
Sep 14 10:48:39.996887 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:39.996863    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z6bgb"
Sep 14 10:48:41.001144 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:41.001106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv" event={"ID":"186e7cba-cf31-41e6-862d-29b8e1156533","Type":"ContainerStarted","Data":"61543b15326f1a6871bc27e11f2aa500442ef485ab83c53ab45aff932104176e"}
Sep 14 10:48:41.019065 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:41.019021    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf5vv" podStartSLOduration=3.667269642 podStartE2EDuration="25.019002873s" totalImagesPullingTime="21.351733231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:41.018610156 +0000 UTC m=+25.703294549" watchObservedRunningTime="2026-09-14 10:48:41.019002873 +0000 UTC m=+25.703687273"
Sep 14 10:48:41.796414 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:41.796379    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:41.796580 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:41.796427    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:41.796580 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:41.796427    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:41.796580 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:41.796513    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:41.796766 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:41.796660    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:41.796766 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:41.796755    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:43.007651 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:43.007433    2561 generic.go:356] "Generic (PLEG): container finished" podID="6db78d45-a197-4361-9e83-fc3929395f94" containerID="e28eb05fae4329382f4f762cf6945da01694378f1b6fcd4e0d4a0aadf515b861" exitCode=0
Sep 14 10:48:43.008067 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:43.007517    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbbc4" event={"ID":"6db78d45-a197-4361-9e83-fc3929395f94","Type":"ContainerDied","Data":"e28eb05fae4329382f4f762cf6945da01694378f1b6fcd4e0d4a0aadf515b861"}
Sep 14 10:48:43.011807 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:43.011776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt" event={"ID":"7bb39c20-00eb-45b3-9fd7-deb3c2040e16","Type":"ContainerStarted","Data":"c583d3a0074cf9ebcfde2aa0ae084b1fa0a8a28c4a882b04c362a2658c1085e4"}
Sep 14 10:48:43.012020 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:43.011999    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:43.012094 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:43.012061    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:43.012148 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:43.012109    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:43.026559 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:43.026541    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:43.026643 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:43.026629    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:48:43.057532 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:43.057497    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt" podStartSLOduration=8.336575598 podStartE2EDuration="27.057485554s" totalImagesPullingTime="18.720909956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:43.055967247 +0000 UTC m=+27.740651635" watchObservedRunningTime="2026-09-14 10:48:43.057485554 +0000 UTC m=+27.742169942"
Sep 14 10:48:43.796904 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:43.796861    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:43.797171 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:43.796861    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:43.797171 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:43.796996    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:43.797171 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:43.797035    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:43.797171 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:43.797098    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:43.797379 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:43.797176    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:44.437694 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:44.437624    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5j9jp"]
Sep 14 10:48:44.438239 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:44.437733    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:44.438239 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:44.437817    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:44.441414 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:44.441388    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-67w9c"]
Sep 14 10:48:44.441517 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:44.441492    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:44.441632 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:44.441606    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:44.441885 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:44.441864    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l5pvm"]
Sep 14 10:48:44.441995 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:44.441929    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:44.442051 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:44.442002    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:45.018439 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:45.018401    2561 generic.go:356] "Generic (PLEG): container finished" podID="6db78d45-a197-4361-9e83-fc3929395f94" containerID="9eca786c032c84573dec43413f6059ceaf0a43a69e769abc7a44f710b247f3b2" exitCode=0
Sep 14 10:48:45.018637 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:45.018482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbbc4" event={"ID":"6db78d45-a197-4361-9e83-fc3929395f94","Type":"ContainerDied","Data":"9eca786c032c84573dec43413f6059ceaf0a43a69e769abc7a44f710b247f3b2"}
Sep 14 10:48:45.798588 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:45.798508    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:45.799096 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:45.798644    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:45.799096 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:45.798692    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:45.799096 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:45.798833    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:46.797447 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:46.797300    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:46.797563 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:46.797531    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:47.025243 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:47.025116    2561 generic.go:356] "Generic (PLEG): container finished" podID="6db78d45-a197-4361-9e83-fc3929395f94" containerID="236e755fd56ef0c6d99aaea3ff6f6ea7f084b873486e794b572cff8bda065044" exitCode=0
Sep 14 10:48:47.025243 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:47.025187    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbbc4" event={"ID":"6db78d45-a197-4361-9e83-fc3929395f94","Type":"ContainerDied","Data":"236e755fd56ef0c6d99aaea3ff6f6ea7f084b873486e794b572cff8bda065044"}
Sep 14 10:48:47.796543 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:47.796511    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:47.796761 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:47.796518    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:47.796761 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:47.796656    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67w9c" podUID="b3d78350-bf09-4796-9e47-d39ec74fde78"
Sep 14 10:48:47.796761 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:47.796700    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5j9jp" podUID="659457bf-d253-49e8-becb-fddc72d02184"
Sep 14 10:48:48.796659 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:48.796628    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:48.797142 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:48.796749    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5pvm" podUID="07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029"
Sep 14 10:48:49.504602 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.504571    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:49.504779 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:49.504735    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:49.504844 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:49.504810    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs podName:b3d78350-bf09-4796-9e47-d39ec74fde78 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:21.504790202 +0000 UTC m=+66.189474576 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs") pod "network-metrics-daemon-67w9c" (UID: "b3d78350-bf09-4796-9e47-d39ec74fde78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:48:49.605625 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.605590    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:49.605795 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.605646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6pm\" (UniqueName: \"kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm\") pod \"network-check-target-5j9jp\" (UID: \"659457bf-d253-49e8-becb-fddc72d02184\") " pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:49.605795 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:49.605748    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:49.605881 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:49.605817    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:48:49.605881 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:49.605835    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:48:49.605881 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:49.605849    2561 projected.go:196] Error preparing data for projected volume kube-api-access-9j6pm for pod openshift-network-diagnostics/network-check-target-5j9jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:49.606007 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:49.605821    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret podName:07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:21.605805736 +0000 UTC m=+66.290490103 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret") pod "global-pull-secret-syncer-l5pvm" (UID: "07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:48:49.606007 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:49.605924    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm podName:659457bf-d253-49e8-becb-fddc72d02184 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:21.605898584 +0000 UTC m=+66.290582961 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9j6pm" (UniqueName: "kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm") pod "network-check-target-5j9jp" (UID: "659457bf-d253-49e8-becb-fddc72d02184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:48:49.626513 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.626452    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeReady"
Sep 14 10:48:49.626620 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.626567    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 14 10:48:49.661763 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.661738    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-694df55669-jswp6"]
Sep 14 10:48:49.666647 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.666629    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.670193 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.670152    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 14 10:48:49.670193 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.670170    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 14 10:48:49.670375 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.670209    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 14 10:48:49.670758 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.670735    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xzrxf\""
Sep 14 10:48:49.676322 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.676306    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lbplh"]
Sep 14 10:48:49.677308 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.677289    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 14 10:48:49.679903 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.679887    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m77xr"]
Sep 14 10:48:49.680005 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.679986    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:49.683799 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.683454    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 14 10:48:49.683799 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.683460    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 14 10:48:49.683799 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.683462    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qfwcj\""
Sep 14 10:48:49.683799 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.683628    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-694df55669-jswp6"]
Sep 14 10:48:49.683799 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.683749    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m77xr"
Sep 14 10:48:49.688159 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.688114    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 14 10:48:49.688392 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.688228    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5cjtt\""
Sep 14 10:48:49.688392 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.688325    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 14 10:48:49.688512 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.688391    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 14 10:48:49.691358 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.691333    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lbplh"]
Sep 14 10:48:49.693227 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.693204    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m77xr"]
Sep 14 10:48:49.796958 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.796932    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:48:49.797416 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.796935    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:48:49.799839 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.799819    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 10:48:49.799839 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.799828    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 10:48:49.800192 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.800174    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bckh8\""
Sep 14 10:48:49.800292 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.800196    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fkf2p\""
Sep 14 10:48:49.800292 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.800174    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 10:48:49.806934 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.806912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b84cb0e-90da-4eee-85db-275a9795abca-cert\") pod \"ingress-canary-m77xr\" (UID: \"5b84cb0e-90da-4eee-85db-275a9795abca\") " pod="openshift-ingress-canary/ingress-canary-m77xr"
Sep 14 10:48:49.807048 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.806967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-ca-trust-extracted\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.807048 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.806993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2rb4\" (UniqueName: \"kubernetes.io/projected/5b84cb0e-90da-4eee-85db-275a9795abca-kube-api-access-v2rb4\") pod \"ingress-canary-m77xr\" (UID: \"5b84cb0e-90da-4eee-85db-275a9795abca\") " pod="openshift-ingress-canary/ingress-canary-m77xr"
Sep 14 10:48:49.807048 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.807009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90f711db-6849-4eb1-9ed2-85b410b8765e-tmp-dir\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:49.807048 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.807033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-installation-pull-secrets\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.807262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.807090    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-trusted-ca\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.807262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.807170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftsqv\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-kube-api-access-ftsqv\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.807262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.807238    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-image-registry-private-configuration\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.807418 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.807264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.807418 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.807290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:49.807418 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.807313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txclx\" (UniqueName: \"kubernetes.io/projected/90f711db-6849-4eb1-9ed2-85b410b8765e-kube-api-access-txclx\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:49.807418 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.807339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-bound-sa-token\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.807418 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.807381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90f711db-6849-4eb1-9ed2-85b410b8765e-config-volume\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:49.807418 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.807409    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-certificates\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.908766 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.908672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2rb4\" (UniqueName: \"kubernetes.io/projected/5b84cb0e-90da-4eee-85db-275a9795abca-kube-api-access-v2rb4\") pod \"ingress-canary-m77xr\" (UID: \"5b84cb0e-90da-4eee-85db-275a9795abca\") " pod="openshift-ingress-canary/ingress-canary-m77xr"
Sep 14 10:48:49.908766 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.908738    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90f711db-6849-4eb1-9ed2-85b410b8765e-tmp-dir\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:49.908983 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.908774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-installation-pull-secrets\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.908983 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.908801    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-trusted-ca\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.908983 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.908842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftsqv\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-kube-api-access-ftsqv\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.909122 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.909031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-image-registry-private-configuration\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.909122 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.909052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.909122 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.909072    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:49.909122 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.909087    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txclx\" (UniqueName: \"kubernetes.io/projected/90f711db-6849-4eb1-9ed2-85b410b8765e-kube-api-access-txclx\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:49.909122 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.909105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-bound-sa-token\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.909357 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.909126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90f711db-6849-4eb1-9ed2-85b410b8765e-config-volume\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:49.909357 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.909144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-certificates\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.909357 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.909171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b84cb0e-90da-4eee-85db-275a9795abca-cert\") pod \"ingress-canary-m77xr\" (UID: \"5b84cb0e-90da-4eee-85db-275a9795abca\") " pod="openshift-ingress-canary/ingress-canary-m77xr"
Sep 14 10:48:49.909357 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.909175    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90f711db-6849-4eb1-9ed2-85b410b8765e-tmp-dir\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:49.909357 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.909195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-ca-trust-extracted\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.909595 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.909409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-ca-trust-extracted\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.909595 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:49.909529    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 10:48:49.909595 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:49.909545    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694df55669-jswp6: secret "image-registry-tls" not found
Sep 14 10:48:49.909595 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:49.909585    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 10:48:49.909795 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:49.909599    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls podName:396c0cfd-285d-4d44-9260-ac4c7d3fffb7 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:50.409579591 +0000 UTC m=+35.094263971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls") pod "image-registry-694df55669-jswp6" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7") : secret "image-registry-tls" not found
Sep 14 10:48:49.909795 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:49.909655    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls podName:90f711db-6849-4eb1-9ed2-85b410b8765e nodeName:}" failed. No retries permitted until 2026-09-14 10:48:50.409636256 +0000 UTC m=+35.094320628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls") pod "dns-default-lbplh" (UID: "90f711db-6849-4eb1-9ed2-85b410b8765e") : secret "dns-default-metrics-tls" not found
Sep 14 10:48:49.909921 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.909807    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-trusted-ca\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.910140 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.910071    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90f711db-6849-4eb1-9ed2-85b410b8765e-config-volume\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:49.910292 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.910268    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-certificates\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.913475 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.913450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-installation-pull-secrets\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.913582 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.913454    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b84cb0e-90da-4eee-85db-275a9795abca-cert\") pod \"ingress-canary-m77xr\" (UID: \"5b84cb0e-90da-4eee-85db-275a9795abca\") " pod="openshift-ingress-canary/ingress-canary-m77xr"
Sep 14 10:48:49.913582 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.913506    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-image-registry-private-configuration\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.918870 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.918800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txclx\" (UniqueName: \"kubernetes.io/projected/90f711db-6849-4eb1-9ed2-85b410b8765e-kube-api-access-txclx\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:49.918870 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.918852    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-bound-sa-token\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:49.919015 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.918936    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2rb4\" (UniqueName: \"kubernetes.io/projected/5b84cb0e-90da-4eee-85db-275a9795abca-kube-api-access-v2rb4\") pod \"ingress-canary-m77xr\" (UID: \"5b84cb0e-90da-4eee-85db-275a9795abca\") " pod="openshift-ingress-canary/ingress-canary-m77xr"
Sep 14 10:48:49.919015 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:49.918992    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftsqv\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-kube-api-access-ftsqv\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:50.000959 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:50.000924    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m77xr"
Sep 14 10:48:50.413374 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:50.413332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:50.413562 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:50.413387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:50.413562 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:50.413491    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 10:48:50.413562 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:50.413496    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 10:48:50.413562 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:50.413519    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694df55669-jswp6: secret "image-registry-tls" not found
Sep 14 10:48:50.413562 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:50.413556    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls podName:90f711db-6849-4eb1-9ed2-85b410b8765e nodeName:}" failed. No retries permitted until 2026-09-14 10:48:51.413539529 +0000 UTC m=+36.098223895 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls") pod "dns-default-lbplh" (UID: "90f711db-6849-4eb1-9ed2-85b410b8765e") : secret "dns-default-metrics-tls" not found
Sep 14 10:48:50.413863 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:50.413574    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls podName:396c0cfd-285d-4d44-9260-ac4c7d3fffb7 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:51.4135637 +0000 UTC m=+36.098248071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls") pod "image-registry-694df55669-jswp6" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7") : secret "image-registry-tls" not found
Sep 14 10:48:50.796459 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:50.796372    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:48:50.799382 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:50.799360    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 10:48:51.328186 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:51.328029    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m77xr"]
Sep 14 10:48:51.334130 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:48:51.334105    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b84cb0e_90da_4eee_85db_275a9795abca.slice/crio-de928221e19647c409fd78ff19fbf7b9d607dde4cc6e5b947b52c4c118bebff2 WatchSource:0}: Error finding container de928221e19647c409fd78ff19fbf7b9d607dde4cc6e5b947b52c4c118bebff2: Status 404 returned error can't find the container with id de928221e19647c409fd78ff19fbf7b9d607dde4cc6e5b947b52c4c118bebff2
Sep 14 10:48:51.421834 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:51.421811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:51.421967 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:51.421843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:51.421967 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:51.421949    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 10:48:51.422067 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:51.421974    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 10:48:51.422067 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:51.421992    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694df55669-jswp6: secret "image-registry-tls" not found
Sep 14 10:48:51.422067 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:51.422000    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls podName:90f711db-6849-4eb1-9ed2-85b410b8765e nodeName:}" failed. No retries permitted until 2026-09-14 10:48:53.42198657 +0000 UTC m=+38.106670939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls") pod "dns-default-lbplh" (UID: "90f711db-6849-4eb1-9ed2-85b410b8765e") : secret "dns-default-metrics-tls" not found
Sep 14 10:48:51.422067 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:51.422055    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls podName:396c0cfd-285d-4d44-9260-ac4c7d3fffb7 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:53.422026621 +0000 UTC m=+38.106711008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls") pod "image-registry-694df55669-jswp6" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7") : secret "image-registry-tls" not found
Sep 14 10:48:52.040398 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:52.040360    2561 generic.go:356] "Generic (PLEG): container finished" podID="6db78d45-a197-4361-9e83-fc3929395f94" containerID="aec2f67739147256a81118cbfe563d9a582de2ebc069d96ef70b2b9acb6c8422" exitCode=0
Sep 14 10:48:52.041057 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:52.040426    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbbc4" event={"ID":"6db78d45-a197-4361-9e83-fc3929395f94","Type":"ContainerDied","Data":"aec2f67739147256a81118cbfe563d9a582de2ebc069d96ef70b2b9acb6c8422"}
Sep 14 10:48:52.041904 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:52.041877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m77xr" event={"ID":"5b84cb0e-90da-4eee-85db-275a9795abca","Type":"ContainerStarted","Data":"de928221e19647c409fd78ff19fbf7b9d607dde4cc6e5b947b52c4c118bebff2"}
Sep 14 10:48:53.048616 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:53.048573    2561 generic.go:356] "Generic (PLEG): container finished" podID="6db78d45-a197-4361-9e83-fc3929395f94" containerID="eab67076f25f822caab94f2bc3f069886d0b6716df991e823b5183637569963f" exitCode=0
Sep 14 10:48:53.049152 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:53.048642    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbbc4" event={"ID":"6db78d45-a197-4361-9e83-fc3929395f94","Type":"ContainerDied","Data":"eab67076f25f822caab94f2bc3f069886d0b6716df991e823b5183637569963f"}
Sep 14 10:48:53.437207 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:53.437174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:53.437207 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:53.437213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:53.437451 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:53.437344    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 10:48:53.437451 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:53.437367    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694df55669-jswp6: secret "image-registry-tls" not found
Sep 14 10:48:53.437451 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:53.437380    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 10:48:53.437451 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:53.437430    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls podName:90f711db-6849-4eb1-9ed2-85b410b8765e nodeName:}" failed. No retries permitted until 2026-09-14 10:48:57.437415915 +0000 UTC m=+42.122100286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls") pod "dns-default-lbplh" (UID: "90f711db-6849-4eb1-9ed2-85b410b8765e") : secret "dns-default-metrics-tls" not found
Sep 14 10:48:53.437451 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:53.437446    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls podName:396c0cfd-285d-4d44-9260-ac4c7d3fffb7 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:57.437439382 +0000 UTC m=+42.122123749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls") pod "image-registry-694df55669-jswp6" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7") : secret "image-registry-tls" not found
Sep 14 10:48:54.054197 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:54.054167    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbbc4" event={"ID":"6db78d45-a197-4361-9e83-fc3929395f94","Type":"ContainerStarted","Data":"73a86fa2a3fb09ae01807f55193bb2511e7a9b7f2e071d8cba766c8c09242a5d"}
Sep 14 10:48:54.077558 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:54.077512    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qbbc4" podStartSLOduration=8.196294345 podStartE2EDuration="38.077491384s" totalImagesPullingTime="29.881197039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:54.0762133 +0000 UTC m=+38.760897679" watchObservedRunningTime="2026-09-14 10:48:54.077491384 +0000 UTC m=+38.762175800"
Sep 14 10:48:54.229139 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:54.229108    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z6bgb_c0aa6e02-ed99-41ac-82a1-7b9a7ab93993/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:54.324569 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:54.324520    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z6bgb_c0aa6e02-ed99-41ac-82a1-7b9a7ab93993/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:54.425363 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:54.425316    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z6bgb_c0aa6e02-ed99-41ac-82a1-7b9a7ab93993/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:55.058269 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:55.058232    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m77xr" event={"ID":"5b84cb0e-90da-4eee-85db-275a9795abca","Type":"ContainerStarted","Data":"2f531b45d423924d1e0c677c4c065de01710a4ef16c354d8e80701916440d852"}
Sep 14 10:48:55.075000 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:55.074959    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m77xr" podStartSLOduration=3.236584699 podStartE2EDuration="6.074945226s" totalImagesPullingTime="2.838360527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:55.074863768 +0000 UTC m=+39.759548160" watchObservedRunningTime="2026-09-14 10:48:55.074945226 +0000 UTC m=+39.759629614"
Sep 14 10:48:57.464899 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:57.464856    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:48:57.464899 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:48:57.464902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:48:57.465302 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:57.465011    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 10:48:57.465302 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:57.465013    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 10:48:57.465302 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:57.465036    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694df55669-jswp6: secret "image-registry-tls" not found
Sep 14 10:48:57.465302 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:57.465064    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls podName:90f711db-6849-4eb1-9ed2-85b410b8765e nodeName:}" failed. No retries permitted until 2026-09-14 10:49:05.465051227 +0000 UTC m=+50.149735599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls") pod "dns-default-lbplh" (UID: "90f711db-6849-4eb1-9ed2-85b410b8765e") : secret "dns-default-metrics-tls" not found
Sep 14 10:48:57.465302 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:48:57.465077    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls podName:396c0cfd-285d-4d44-9260-ac4c7d3fffb7 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:05.465071745 +0000 UTC m=+50.149756112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls") pod "image-registry-694df55669-jswp6" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7") : secret "image-registry-tls" not found
Sep 14 10:49:00.696376 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:00.696349    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z6bgb_c0aa6e02-ed99-41ac-82a1-7b9a7ab93993/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:00.827408 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:00.827380    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z6bgb_c0aa6e02-ed99-41ac-82a1-7b9a7ab93993/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:00.926990 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:00.926954    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z6bgb_c0aa6e02-ed99-41ac-82a1-7b9a7ab93993/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:01.027014 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:01.026920    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z6bgb_c0aa6e02-ed99-41ac-82a1-7b9a7ab93993/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:01.151575 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:01.151533    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z6bgb_c0aa6e02-ed99-41ac-82a1-7b9a7ab93993/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:01.717749 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:01.717723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z6bgb_c0aa6e02-ed99-41ac-82a1-7b9a7ab93993/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:01.882440 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:01.882413    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z6bgb_c0aa6e02-ed99-41ac-82a1-7b9a7ab93993/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:05.519292 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:05.519254    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:49:05.519292 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:05.519292    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:49:05.519690 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:05.519383    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 10:49:05.519690 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:05.519388    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 10:49:05.519690 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:05.519403    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694df55669-jswp6: secret "image-registry-tls" not found
Sep 14 10:49:05.519690 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:05.519445    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls podName:90f711db-6849-4eb1-9ed2-85b410b8765e nodeName:}" failed. No retries permitted until 2026-09-14 10:49:21.519430304 +0000 UTC m=+66.204114675 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls") pod "dns-default-lbplh" (UID: "90f711db-6849-4eb1-9ed2-85b410b8765e") : secret "dns-default-metrics-tls" not found
Sep 14 10:49:05.519690 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:05.519457    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls podName:396c0cfd-285d-4d44-9260-ac4c7d3fffb7 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:21.519451894 +0000 UTC m=+66.204136261 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls") pod "image-registry-694df55669-jswp6" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7") : secret "image-registry-tls" not found
Sep 14 10:49:15.040409 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:15.040381    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thtt"
Sep 14 10:49:21.523680 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.523642    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:49:21.523680 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.523685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:49:21.524249 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.523703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:49:21.524249 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:21.523826    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 10:49:21.524249 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:21.523893    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls podName:90f711db-6849-4eb1-9ed2-85b410b8765e nodeName:}" failed. No retries permitted until 2026-09-14 10:49:53.523867967 +0000 UTC m=+98.208552334 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls") pod "dns-default-lbplh" (UID: "90f711db-6849-4eb1-9ed2-85b410b8765e") : secret "dns-default-metrics-tls" not found
Sep 14 10:49:21.524249 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:21.523824    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 10:49:21.524249 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:21.523929    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694df55669-jswp6: secret "image-registry-tls" not found
Sep 14 10:49:21.524249 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:21.523991    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls podName:396c0cfd-285d-4d44-9260-ac4c7d3fffb7 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:53.523977206 +0000 UTC m=+98.208661573 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls") pod "image-registry-694df55669-jswp6" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7") : secret "image-registry-tls" not found
Sep 14 10:49:21.526595 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.526582    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 10:49:21.534029 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:21.534009    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 14 10:49:21.534108 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:21.534060    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs podName:b3d78350-bf09-4796-9e47-d39ec74fde78 nodeName:}" failed. No retries permitted until 2026-09-14 10:50:25.534048892 +0000 UTC m=+130.218733264 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs") pod "network-metrics-daemon-67w9c" (UID: "b3d78350-bf09-4796-9e47-d39ec74fde78") : secret "metrics-daemon-secret" not found
Sep 14 10:49:21.624294 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.624266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:49:21.624425 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.624306    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6pm\" (UniqueName: \"kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm\") pod \"network-check-target-5j9jp\" (UID: \"659457bf-d253-49e8-becb-fddc72d02184\") " pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:49:21.627006 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.626988    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 10:49:21.627104 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.627006    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 10:49:21.637677 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.637655    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 10:49:21.638644 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.638628    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029-original-pull-secret\") pod \"global-pull-secret-syncer-l5pvm\" (UID: \"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029\") " pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:49:21.648279 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.648254    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j6pm\" (UniqueName: \"kubernetes.io/projected/659457bf-d253-49e8-becb-fddc72d02184-kube-api-access-9j6pm\") pod \"network-check-target-5j9jp\" (UID: \"659457bf-d253-49e8-becb-fddc72d02184\") " pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:49:21.706470 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.706446    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5pvm"
Sep 14 10:49:21.918999 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.918966    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fkf2p\""
Sep 14 10:49:21.927165 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.927145    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:49:21.975450 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:21.975405    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l5pvm"]
Sep 14 10:49:21.979076 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:49:21.979049    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07dfb8b4_5bb4_4e2b_ab85_7b4dd7872029.slice/crio-c15a10f52971072d0bcc2b6b1da072443422b92c8c925e28742ff80df8cf71c4 WatchSource:0}: Error finding container c15a10f52971072d0bcc2b6b1da072443422b92c8c925e28742ff80df8cf71c4: Status 404 returned error can't find the container with id c15a10f52971072d0bcc2b6b1da072443422b92c8c925e28742ff80df8cf71c4
Sep 14 10:49:22.120462 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:22.120432    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l5pvm" event={"ID":"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029","Type":"ContainerStarted","Data":"c15a10f52971072d0bcc2b6b1da072443422b92c8c925e28742ff80df8cf71c4"}
Sep 14 10:49:22.215762 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:22.215681    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5j9jp"]
Sep 14 10:49:22.218447 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:49:22.218424    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659457bf_d253_49e8_becb_fddc72d02184.slice/crio-c237f8bfd74678eaea9d9a6d953391a038e9fef9828774735630fb9690cb99c7 WatchSource:0}: Error finding container c237f8bfd74678eaea9d9a6d953391a038e9fef9828774735630fb9690cb99c7: Status 404 returned error can't find the container with id c237f8bfd74678eaea9d9a6d953391a038e9fef9828774735630fb9690cb99c7
Sep 14 10:49:23.124790 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:23.124747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5j9jp" event={"ID":"659457bf-d253-49e8-becb-fddc72d02184","Type":"ContainerStarted","Data":"c237f8bfd74678eaea9d9a6d953391a038e9fef9828774735630fb9690cb99c7"}
Sep 14 10:49:27.135993 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:27.135958    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5j9jp" event={"ID":"659457bf-d253-49e8-becb-fddc72d02184","Type":"ContainerStarted","Data":"d16124f7a3544a0177775bc23c443ff292ba4b9c36af7e65cf67c632e76f9e66"}
Sep 14 10:49:27.136470 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:27.136030    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:49:27.137501 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:27.137478    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l5pvm" event={"ID":"07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029","Type":"ContainerStarted","Data":"45dc48dfd5ff6261d632a8290fdde7765a25a9d7455941fa721637911c4e60fa"}
Sep 14 10:49:27.151811 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:27.151771    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5j9jp" podStartSLOduration=66.549747609 podStartE2EDuration="1m11.151756974s" totalImagesPullingTime="4.602009365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:27.151627965 +0000 UTC m=+71.836312354" watchObservedRunningTime="2026-09-14 10:49:27.151756974 +0000 UTC m=+71.836441379"
Sep 14 10:49:27.169849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:27.169812    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l5pvm" podStartSLOduration=66.324092089 podStartE2EDuration="1m11.169801491s" totalImagesPullingTime="4.845709402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:27.168862192 +0000 UTC m=+71.853546581" watchObservedRunningTime="2026-09-14 10:49:27.169801491 +0000 UTC m=+71.854485880"
Sep 14 10:49:44.067056 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.067023    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"]
Sep 14 10:49:44.071048 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.071029    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-fxv4b"]
Sep 14 10:49:44.071174 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.071153    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"
Sep 14 10:49:44.073578 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.073560    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.074082 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.074061    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 14 10:49:44.074197 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.074099    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 14 10:49:44.074197 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.074140    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 14 10:49:44.074197 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.074102    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ckgnm\""
Sep 14 10:49:44.076098 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.076078    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 14 10:49:44.076098 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.076088    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 14 10:49:44.076264 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.076158    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 14 10:49:44.076264 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.076174    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 14 10:49:44.076372 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.076344    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bzgnc\""
Sep 14 10:49:44.081610 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.081590    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"]
Sep 14 10:49:44.082552 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.082533    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-fxv4b"]
Sep 14 10:49:44.083413 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.083395    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 14 10:49:44.178385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.178361    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-64hq9\" (UID: \"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"
Sep 14 10:49:44.178499 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.178389    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrtst\" (UniqueName: \"kubernetes.io/projected/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-kube-api-access-rrtst\") pod \"cluster-samples-operator-86499c89c4-64hq9\" (UID: \"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"
Sep 14 10:49:44.178499 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.178406    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/834e2d7c-b518-4988-a840-384226d6233d-trusted-ca\") pod \"console-operator-696884fb9c-fxv4b\" (UID: \"834e2d7c-b518-4988-a840-384226d6233d\") " pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.178499 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.178445    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xlhr\" (UniqueName: \"kubernetes.io/projected/834e2d7c-b518-4988-a840-384226d6233d-kube-api-access-4xlhr\") pod \"console-operator-696884fb9c-fxv4b\" (UID: \"834e2d7c-b518-4988-a840-384226d6233d\") " pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.178499 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.178467    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/834e2d7c-b518-4988-a840-384226d6233d-config\") pod \"console-operator-696884fb9c-fxv4b\" (UID: \"834e2d7c-b518-4988-a840-384226d6233d\") " pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.178658 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.178537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/834e2d7c-b518-4988-a840-384226d6233d-serving-cert\") pod \"console-operator-696884fb9c-fxv4b\" (UID: \"834e2d7c-b518-4988-a840-384226d6233d\") " pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.279235 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.279211    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/834e2d7c-b518-4988-a840-384226d6233d-config\") pod \"console-operator-696884fb9c-fxv4b\" (UID: \"834e2d7c-b518-4988-a840-384226d6233d\") " pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.279345 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.279267    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/834e2d7c-b518-4988-a840-384226d6233d-serving-cert\") pod \"console-operator-696884fb9c-fxv4b\" (UID: \"834e2d7c-b518-4988-a840-384226d6233d\") " pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.279345 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.279288    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-64hq9\" (UID: \"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"
Sep 14 10:49:44.279345 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.279305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrtst\" (UniqueName: \"kubernetes.io/projected/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-kube-api-access-rrtst\") pod \"cluster-samples-operator-86499c89c4-64hq9\" (UID: \"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"
Sep 14 10:49:44.279345 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.279321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/834e2d7c-b518-4988-a840-384226d6233d-trusted-ca\") pod \"console-operator-696884fb9c-fxv4b\" (UID: \"834e2d7c-b518-4988-a840-384226d6233d\") " pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.279345 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.279345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xlhr\" (UniqueName: \"kubernetes.io/projected/834e2d7c-b518-4988-a840-384226d6233d-kube-api-access-4xlhr\") pod \"console-operator-696884fb9c-fxv4b\" (UID: \"834e2d7c-b518-4988-a840-384226d6233d\") " pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.279596 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:44.279439    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 10:49:44.279596 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:44.279505    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls podName:2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:44.779485367 +0000 UTC m=+89.464169749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-64hq9" (UID: "2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3") : secret "samples-operator-tls" not found
Sep 14 10:49:44.279959 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.279932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/834e2d7c-b518-4988-a840-384226d6233d-config\") pod \"console-operator-696884fb9c-fxv4b\" (UID: \"834e2d7c-b518-4988-a840-384226d6233d\") " pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.280186 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.280168    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/834e2d7c-b518-4988-a840-384226d6233d-trusted-ca\") pod \"console-operator-696884fb9c-fxv4b\" (UID: \"834e2d7c-b518-4988-a840-384226d6233d\") " pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.281502 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.281485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/834e2d7c-b518-4988-a840-384226d6233d-serving-cert\") pod \"console-operator-696884fb9c-fxv4b\" (UID: \"834e2d7c-b518-4988-a840-384226d6233d\") " pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.289952 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.289933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrtst\" (UniqueName: \"kubernetes.io/projected/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-kube-api-access-rrtst\") pod \"cluster-samples-operator-86499c89c4-64hq9\" (UID: \"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"
Sep 14 10:49:44.290154 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.290137    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xlhr\" (UniqueName: \"kubernetes.io/projected/834e2d7c-b518-4988-a840-384226d6233d-kube-api-access-4xlhr\") pod \"console-operator-696884fb9c-fxv4b\" (UID: \"834e2d7c-b518-4988-a840-384226d6233d\") " pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.388055 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.388032    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:44.656605 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.656581    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-fxv4b"]
Sep 14 10:49:44.658820 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:49:44.658788    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834e2d7c_b518_4988_a840_384226d6233d.slice/crio-c727ed02cd536096fe24a3a59647bb7aaaa361b4b37ed4ff862eb4ca27933d6f WatchSource:0}: Error finding container c727ed02cd536096fe24a3a59647bb7aaaa361b4b37ed4ff862eb4ca27933d6f: Status 404 returned error can't find the container with id c727ed02cd536096fe24a3a59647bb7aaaa361b4b37ed4ff862eb4ca27933d6f
Sep 14 10:49:44.782094 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:44.782066    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-64hq9\" (UID: \"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"
Sep 14 10:49:44.782217 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:44.782193    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 10:49:44.782279 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:44.782273    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls podName:2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:45.782252423 +0000 UTC m=+90.466936792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-64hq9" (UID: "2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3") : secret "samples-operator-tls" not found
Sep 14 10:49:45.183301 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:45.183265    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b" event={"ID":"834e2d7c-b518-4988-a840-384226d6233d","Type":"ContainerStarted","Data":"c727ed02cd536096fe24a3a59647bb7aaaa361b4b37ed4ff862eb4ca27933d6f"}
Sep 14 10:49:45.789213 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:45.789174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-64hq9\" (UID: \"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"
Sep 14 10:49:45.789393 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:45.789348    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 10:49:45.789442 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:45.789428    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls podName:2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:47.789411372 +0000 UTC m=+92.474095744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-64hq9" (UID: "2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3") : secret "samples-operator-tls" not found
Sep 14 10:49:47.189263 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:47.189229    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/0.log"
Sep 14 10:49:47.189652 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:47.189276    2561 generic.go:356] "Generic (PLEG): container finished" podID="834e2d7c-b518-4988-a840-384226d6233d" containerID="2083e16518aa11472d2e4e6b2884f51c3b5e29b16be408b069a9913956e21a94" exitCode=255
Sep 14 10:49:47.189652 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:47.189305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b" event={"ID":"834e2d7c-b518-4988-a840-384226d6233d","Type":"ContainerDied","Data":"2083e16518aa11472d2e4e6b2884f51c3b5e29b16be408b069a9913956e21a94"}
Sep 14 10:49:47.189652 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:47.189529    2561 scope.go:117] "RemoveContainer" containerID="2083e16518aa11472d2e4e6b2884f51c3b5e29b16be408b069a9913956e21a94"
Sep 14 10:49:47.802783 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:47.802756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-64hq9\" (UID: \"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"
Sep 14 10:49:47.802925 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:47.802863    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 10:49:47.802925 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:47.802908    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls podName:2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:51.802896535 +0000 UTC m=+96.487580906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-64hq9" (UID: "2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3") : secret "samples-operator-tls" not found
Sep 14 10:49:48.192940 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.192915    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 10:49:48.193315 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.193301    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/0.log"
Sep 14 10:49:48.193375 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.193334    2561 generic.go:356] "Generic (PLEG): container finished" podID="834e2d7c-b518-4988-a840-384226d6233d" containerID="f3be9cb9a4ad87c6a7f7783a43320592d236546c537a39ee450a8828bab07064" exitCode=255
Sep 14 10:49:48.193410 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.193393    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b" event={"ID":"834e2d7c-b518-4988-a840-384226d6233d","Type":"ContainerDied","Data":"f3be9cb9a4ad87c6a7f7783a43320592d236546c537a39ee450a8828bab07064"}
Sep 14 10:49:48.193454 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.193443    2561 scope.go:117] "RemoveContainer" containerID="2083e16518aa11472d2e4e6b2884f51c3b5e29b16be408b069a9913956e21a94"
Sep 14 10:49:48.193610 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.193592    2561 scope.go:117] "RemoveContainer" containerID="f3be9cb9a4ad87c6a7f7783a43320592d236546c537a39ee450a8828bab07064"
Sep 14 10:49:48.193789 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:48.193770    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-fxv4b_openshift-console-operator(834e2d7c-b518-4988-a840-384226d6233d)\"" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b" podUID="834e2d7c-b518-4988-a840-384226d6233d"
Sep 14 10:49:48.394004 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.393974    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ftztr"]
Sep 14 10:49:48.399478 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.399455    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:48.405016 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.404993    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 14 10:49:48.405141 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.405043    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 14 10:49:48.409195 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.409179    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6n6bp\""
Sep 14 10:49:48.410644 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.410629    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 14 10:49:48.412615 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.412603    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 14 10:49:48.434020 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.433994    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ftztr"]
Sep 14 10:49:48.508167 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.508110    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be8bbde4-d6c1-4267-bf03-c57928bc8e04-crio-socket\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:48.508167 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.508154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be8bbde4-d6c1-4267-bf03-c57928bc8e04-data-volume\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:48.508301 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.508171    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:48.508301 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.508190    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2cd5\" (UniqueName: \"kubernetes.io/projected/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-api-access-w2cd5\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:48.508301 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.508267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:48.609015 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.608989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be8bbde4-d6c1-4267-bf03-c57928bc8e04-crio-socket\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:48.609174 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.609030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be8bbde4-d6c1-4267-bf03-c57928bc8e04-data-volume\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:48.609174 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.609045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:48.609174 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:48.609097    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 10:49:48.609174 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.609109    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be8bbde4-d6c1-4267-bf03-c57928bc8e04-crio-socket\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:48.609174 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:48.609142    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm podName:be8bbde4-d6c1-4267-bf03-c57928bc8e04 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:49.1091293 +0000 UTC m=+93.793813667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ftztr" (UID: "be8bbde4-d6c1-4267-bf03-c57928bc8e04") : configmap "kube-rbac-proxy" not found
Sep 14 10:49:48.609174 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.609165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2cd5\" (UniqueName: \"kubernetes.io/projected/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-api-access-w2cd5\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:48.609402 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.609221    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:48.609402 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:48.609349    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 10:49:48.609402 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.609354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be8bbde4-d6c1-4267-bf03-c57928bc8e04-data-volume\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:48.609489 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:48.609416    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls podName:be8bbde4-d6c1-4267-bf03-c57928bc8e04 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:49.109400629 +0000 UTC m=+93.794084996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ftztr" (UID: "be8bbde4-d6c1-4267-bf03-c57928bc8e04") : secret "insights-runtime-extractor-tls" not found
Sep 14 10:49:48.617863 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:48.617846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2cd5\" (UniqueName: \"kubernetes.io/projected/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-api-access-w2cd5\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:49.113016 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:49.112981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:49.113182 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:49.113030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:49.113182 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:49.113102    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 10:49:49.113182 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:49.113142    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 10:49:49.113182 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:49.113177    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm podName:be8bbde4-d6c1-4267-bf03-c57928bc8e04 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:50.113160939 +0000 UTC m=+94.797845306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ftztr" (UID: "be8bbde4-d6c1-4267-bf03-c57928bc8e04") : configmap "kube-rbac-proxy" not found
Sep 14 10:49:49.113321 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:49.113193    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls podName:be8bbde4-d6c1-4267-bf03-c57928bc8e04 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:50.113186727 +0000 UTC m=+94.797871094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ftztr" (UID: "be8bbde4-d6c1-4267-bf03-c57928bc8e04") : secret "insights-runtime-extractor-tls" not found
Sep 14 10:49:49.197591 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:49.197568    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 10:49:49.197947 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:49.197891    2561 scope.go:117] "RemoveContainer" containerID="f3be9cb9a4ad87c6a7f7783a43320592d236546c537a39ee450a8828bab07064"
Sep 14 10:49:49.198042 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:49.198023    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-fxv4b_openshift-console-operator(834e2d7c-b518-4988-a840-384226d6233d)\"" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b" podUID="834e2d7c-b518-4988-a840-384226d6233d"
Sep 14 10:49:50.121488 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:50.121451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:50.121656 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:50.121501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:50.121656 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:50.121573    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 10:49:50.121656 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:50.121616    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 10:49:50.121656 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:50.121638    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm podName:be8bbde4-d6c1-4267-bf03-c57928bc8e04 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:52.121623453 +0000 UTC m=+96.806307825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ftztr" (UID: "be8bbde4-d6c1-4267-bf03-c57928bc8e04") : configmap "kube-rbac-proxy" not found
Sep 14 10:49:50.121656 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:50.121654    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls podName:be8bbde4-d6c1-4267-bf03-c57928bc8e04 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:52.121644136 +0000 UTC m=+96.806328508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ftztr" (UID: "be8bbde4-d6c1-4267-bf03-c57928bc8e04") : secret "insights-runtime-extractor-tls" not found
Sep 14 10:49:50.449137 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:50.449063    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nr9n4_33c59f54-75fd-4277-86ad-b69ec8f49176/dns-node-resolver/0.log"
Sep 14 10:49:51.646561 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:51.646535    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xmbk7_914ecb92-cd00-4b96-bed0-d0df7d566386/node-ca/0.log"
Sep 14 10:49:51.834386 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:51.834362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-64hq9\" (UID: \"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"
Sep 14 10:49:51.834564 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:51.834501    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 10:49:51.834641 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:51.834584    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls podName:2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:59.834563086 +0000 UTC m=+104.519247457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-64hq9" (UID: "2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3") : secret "samples-operator-tls" not found
Sep 14 10:49:52.046334 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.046261    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m77xr_5b84cb0e-90da-4eee-85db-275a9795abca/serve-healthcheck-canary/0.log"
Sep 14 10:49:52.081933 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.081909    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-smvrh"]
Sep 14 10:49:52.084798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.084784    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-smvrh"
Sep 14 10:49:52.087482 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.087462    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 14 10:49:52.087604 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.087508    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 14 10:49:52.087604 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.087571    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 14 10:49:52.088594 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.088568    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 14 10:49:52.088685 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.088652    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6j4qj\""
Sep 14 10:49:52.093821 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.093802    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-smvrh"]
Sep 14 10:49:52.136456 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.136434    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:52.136565 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.136465    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc-signing-cabundle\") pod \"service-ca-568869fdc5-smvrh\" (UID: \"35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc\") " pod="openshift-service-ca/service-ca-568869fdc5-smvrh"
Sep 14 10:49:52.136565 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.136526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:52.136565 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.136546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc-signing-key\") pod \"service-ca-568869fdc5-smvrh\" (UID: \"35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc\") " pod="openshift-service-ca/service-ca-568869fdc5-smvrh"
Sep 14 10:49:52.136565 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.136561    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbrs\" (UniqueName: \"kubernetes.io/projected/35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc-kube-api-access-kxbrs\") pod \"service-ca-568869fdc5-smvrh\" (UID: \"35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc\") " pod="openshift-service-ca/service-ca-568869fdc5-smvrh"
Sep 14 10:49:52.136707 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:52.136572    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 10:49:52.136707 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:52.136626    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls podName:be8bbde4-d6c1-4267-bf03-c57928bc8e04 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:56.136612153 +0000 UTC m=+100.821296526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ftztr" (UID: "be8bbde4-d6c1-4267-bf03-c57928bc8e04") : secret "insights-runtime-extractor-tls" not found
Sep 14 10:49:52.136707 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:52.136637    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 10:49:52.136707 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:52.136673    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm podName:be8bbde4-d6c1-4267-bf03-c57928bc8e04 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:56.136662595 +0000 UTC m=+100.821346966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ftztr" (UID: "be8bbde4-d6c1-4267-bf03-c57928bc8e04") : configmap "kube-rbac-proxy" not found
Sep 14 10:49:52.237553 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.237534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc-signing-cabundle\") pod \"service-ca-568869fdc5-smvrh\" (UID: \"35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc\") " pod="openshift-service-ca/service-ca-568869fdc5-smvrh"
Sep 14 10:49:52.237650 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.237627    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc-signing-key\") pod \"service-ca-568869fdc5-smvrh\" (UID: \"35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc\") " pod="openshift-service-ca/service-ca-568869fdc5-smvrh"
Sep 14 10:49:52.237693 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.237652    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbrs\" (UniqueName: \"kubernetes.io/projected/35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc-kube-api-access-kxbrs\") pod \"service-ca-568869fdc5-smvrh\" (UID: \"35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc\") " pod="openshift-service-ca/service-ca-568869fdc5-smvrh"
Sep 14 10:49:52.238236 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.238213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc-signing-cabundle\") pod \"service-ca-568869fdc5-smvrh\" (UID: \"35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc\") " pod="openshift-service-ca/service-ca-568869fdc5-smvrh"
Sep 14 10:49:52.239992 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.239967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc-signing-key\") pod \"service-ca-568869fdc5-smvrh\" (UID: \"35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc\") " pod="openshift-service-ca/service-ca-568869fdc5-smvrh"
Sep 14 10:49:52.246913 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.246891    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbrs\" (UniqueName: \"kubernetes.io/projected/35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc-kube-api-access-kxbrs\") pod \"service-ca-568869fdc5-smvrh\" (UID: \"35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc\") " pod="openshift-service-ca/service-ca-568869fdc5-smvrh"
Sep 14 10:49:52.393281 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.393249    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-smvrh"
Sep 14 10:49:52.656470 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:52.656323    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-smvrh"]
Sep 14 10:49:52.658851 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:49:52.658826    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ea7271_e93c_4bf4_817d_9fc0fc5ed3bc.slice/crio-08f8c5a55715fb7dfb0adb158065327d08c8dc18784c32d3aff36245be0a1cf8 WatchSource:0}: Error finding container 08f8c5a55715fb7dfb0adb158065327d08c8dc18784c32d3aff36245be0a1cf8: Status 404 returned error can't find the container with id 08f8c5a55715fb7dfb0adb158065327d08c8dc18784c32d3aff36245be0a1cf8
Sep 14 10:49:53.208344 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:53.208311    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-smvrh" event={"ID":"35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc","Type":"ContainerStarted","Data":"08f8c5a55715fb7dfb0adb158065327d08c8dc18784c32d3aff36245be0a1cf8"}
Sep 14 10:49:53.545306 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:53.545219    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls\") pod \"image-registry-694df55669-jswp6\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") " pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:49:53.545306 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:53.545259    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:49:53.545508 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:53.545406    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 10:49:53.545508 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:53.545471    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls podName:90f711db-6849-4eb1-9ed2-85b410b8765e nodeName:}" failed. No retries permitted until 2026-09-14 10:50:57.545455721 +0000 UTC m=+162.230140092 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls") pod "dns-default-lbplh" (UID: "90f711db-6849-4eb1-9ed2-85b410b8765e") : secret "dns-default-metrics-tls" not found
Sep 14 10:49:53.545508 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:53.545409    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 10:49:53.545508 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:53.545500    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694df55669-jswp6: secret "image-registry-tls" not found
Sep 14 10:49:53.545681 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:53.545552    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls podName:396c0cfd-285d-4d44-9260-ac4c7d3fffb7 nodeName:}" failed. No retries permitted until 2026-09-14 10:50:57.545539961 +0000 UTC m=+162.230224340 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls") pod "image-registry-694df55669-jswp6" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7") : secret "image-registry-tls" not found
Sep 14 10:49:54.388892 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:54.388847    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:54.389357 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:54.388922    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:49:54.389439 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:54.389364    2561 scope.go:117] "RemoveContainer" containerID="f3be9cb9a4ad87c6a7f7783a43320592d236546c537a39ee450a8828bab07064"
Sep 14 10:49:54.389600 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:54.389568    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-fxv4b_openshift-console-operator(834e2d7c-b518-4988-a840-384226d6233d)\"" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b" podUID="834e2d7c-b518-4988-a840-384226d6233d"
Sep 14 10:49:55.215317 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:55.215277    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-smvrh" event={"ID":"35ea7271-e93c-4bf4-817d-9fc0fc5ed3bc","Type":"ContainerStarted","Data":"7673ff1600c80aa45837da08587aabac01d8da99b9a2b0e4a7e7e48947078ead"}
Sep 14 10:49:55.231228 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:55.231181    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-smvrh" podStartSLOduration=1.408916257 podStartE2EDuration="3.231166136s" totalImagesPullingTime="1.822249879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:55.23021696 +0000 UTC m=+99.914901348" watchObservedRunningTime="2026-09-14 10:49:55.231166136 +0000 UTC m=+99.915850519"
Sep 14 10:49:56.164232 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:56.164163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:56.164652 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:56.164290    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:49:56.164652 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:56.164295    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 10:49:56.164652 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:56.164475    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm podName:be8bbde4-d6c1-4267-bf03-c57928bc8e04 nodeName:}" failed. No retries permitted until 2026-09-14 10:50:04.164458596 +0000 UTC m=+108.849142975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ftztr" (UID: "be8bbde4-d6c1-4267-bf03-c57928bc8e04") : configmap "kube-rbac-proxy" not found
Sep 14 10:49:56.164652 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:56.164363    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 10:49:56.164652 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:49:56.164531    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls podName:be8bbde4-d6c1-4267-bf03-c57928bc8e04 nodeName:}" failed. No retries permitted until 2026-09-14 10:50:04.164520555 +0000 UTC m=+108.849204952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ftztr" (UID: "be8bbde4-d6c1-4267-bf03-c57928bc8e04") : secret "insights-runtime-extractor-tls" not found
Sep 14 10:49:58.142812 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:58.142784    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5j9jp"
Sep 14 10:49:59.897654 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:59.897617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-64hq9\" (UID: \"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"
Sep 14 10:49:59.900220 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:59.900193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-64hq9\" (UID: \"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"
Sep 14 10:49:59.981955 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:49:59.981922    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"
Sep 14 10:50:00.253560 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:00.253493    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9"]
Sep 14 10:50:01.232354 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:01.232318    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9" event={"ID":"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3","Type":"ContainerStarted","Data":"d9045a0e62e9d41a2ed63ec35ec3ba292370af0fe8f8f29a0ae968cee3116e06"}
Sep 14 10:50:02.238887 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:02.238854    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9" event={"ID":"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3","Type":"ContainerStarted","Data":"2e496a352af672cf86acf9580d9b9bd71db81f89a7156fc558794535a4ec753d"}
Sep 14 10:50:03.247064 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:03.247028    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9" event={"ID":"2f58d6b4-9bbc-4eb0-9a03-cc6361d0f5d3","Type":"ContainerStarted","Data":"c568dbd7deafd5b0a0facccd93f8df80c22ebcb7c3f7fc177a5c4a4128ed9fed"}
Sep 14 10:50:03.264436 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:03.264394    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-64hq9" podStartSLOduration=17.45260898 podStartE2EDuration="19.264377874s" totalImagesPullingTime="1.811768894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:03.2638076 +0000 UTC m=+107.948492002" watchObservedRunningTime="2026-09-14 10:50:03.264377874 +0000 UTC m=+107.949062289"
Sep 14 10:50:04.236181 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:04.236142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:50:04.236366 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:04.236239    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:50:04.236366 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:04.236277    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 10:50:04.236366 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:04.236350    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls podName:be8bbde4-d6c1-4267-bf03-c57928bc8e04 nodeName:}" failed. No retries permitted until 2026-09-14 10:50:20.236333211 +0000 UTC m=+124.921017593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ftztr" (UID: "be8bbde4-d6c1-4267-bf03-c57928bc8e04") : secret "insights-runtime-extractor-tls" not found
Sep 14 10:50:04.236773 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:04.236756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be8bbde4-d6c1-4267-bf03-c57928bc8e04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:50:06.797535 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:06.797506    2561 scope.go:117] "RemoveContainer" containerID="f3be9cb9a4ad87c6a7f7783a43320592d236546c537a39ee450a8828bab07064"
Sep 14 10:50:07.262054 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:07.262026    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 10:50:07.262235 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:07.262084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b" event={"ID":"834e2d7c-b518-4988-a840-384226d6233d","Type":"ContainerStarted","Data":"721b3f7d7b170e77fc1fb3bb995aa633e8f34a1fafcca427de35d06a39bfc8c0"}
Sep 14 10:50:07.262356 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:07.262337    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:50:07.283741 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:07.281088    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b" podStartSLOduration=21.46114851 podStartE2EDuration="23.281067486s" totalImagesPullingTime="1.819918976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:07.278695492 +0000 UTC m=+111.963379881" watchObservedRunningTime="2026-09-14 10:50:07.281067486 +0000 UTC m=+111.965751876"
Sep 14 10:50:07.478333 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:07.478306    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-fxv4b"
Sep 14 10:50:13.580593 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.580565    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"]
Sep 14 10:50:13.585318 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.583741    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"
Sep 14 10:50:13.590236 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.590215    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 14 10:50:13.591544 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.591524    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 14 10:50:13.591544 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.591538    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 14 10:50:13.591734 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.591568    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 14 10:50:13.606480 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.606457    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"]
Sep 14 10:50:13.667909 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.667882    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-2rk97"]
Sep 14 10:50:13.670920 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.670905    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-2rk97"
Sep 14 10:50:13.673466 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.673445    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 14 10:50:13.673559 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.673488    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 14 10:50:13.673615 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.673598    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9l9mq\""
Sep 14 10:50:13.683182 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.683164    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-2rk97"]
Sep 14 10:50:13.707306 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.707272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbd9819f-4d82-414a-9327-1abbe9d42784-tmp\") pod \"klusterlet-addon-workmgr-6fd5c88574-mfmpw\" (UID: \"bbd9819f-4d82-414a-9327-1abbe9d42784\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"
Sep 14 10:50:13.707413 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.707343    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bbd9819f-4d82-414a-9327-1abbe9d42784-klusterlet-config\") pod \"klusterlet-addon-workmgr-6fd5c88574-mfmpw\" (UID: \"bbd9819f-4d82-414a-9327-1abbe9d42784\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"
Sep 14 10:50:13.707469 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.707450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m56bl\" (UniqueName: \"kubernetes.io/projected/bbd9819f-4d82-414a-9327-1abbe9d42784-kube-api-access-m56bl\") pod \"klusterlet-addon-workmgr-6fd5c88574-mfmpw\" (UID: \"bbd9819f-4d82-414a-9327-1abbe9d42784\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"
Sep 14 10:50:13.808455 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.808428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m56bl\" (UniqueName: \"kubernetes.io/projected/bbd9819f-4d82-414a-9327-1abbe9d42784-kube-api-access-m56bl\") pod \"klusterlet-addon-workmgr-6fd5c88574-mfmpw\" (UID: \"bbd9819f-4d82-414a-9327-1abbe9d42784\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"
Sep 14 10:50:13.808570 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.808470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2sd6\" (UniqueName: \"kubernetes.io/projected/00887c1f-3e0f-4a86-8e54-35e02c9ffa53-kube-api-access-s2sd6\") pod \"downloads-76f99584f4-2rk97\" (UID: \"00887c1f-3e0f-4a86-8e54-35e02c9ffa53\") " pod="openshift-console/downloads-76f99584f4-2rk97"
Sep 14 10:50:13.808570 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.808540    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbd9819f-4d82-414a-9327-1abbe9d42784-tmp\") pod \"klusterlet-addon-workmgr-6fd5c88574-mfmpw\" (UID: \"bbd9819f-4d82-414a-9327-1abbe9d42784\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"
Sep 14 10:50:13.808680 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.808573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bbd9819f-4d82-414a-9327-1abbe9d42784-klusterlet-config\") pod \"klusterlet-addon-workmgr-6fd5c88574-mfmpw\" (UID: \"bbd9819f-4d82-414a-9327-1abbe9d42784\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"
Sep 14 10:50:13.808954 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.808933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbd9819f-4d82-414a-9327-1abbe9d42784-tmp\") pod \"klusterlet-addon-workmgr-6fd5c88574-mfmpw\" (UID: \"bbd9819f-4d82-414a-9327-1abbe9d42784\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"
Sep 14 10:50:13.810964 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.810944    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bbd9819f-4d82-414a-9327-1abbe9d42784-klusterlet-config\") pod \"klusterlet-addon-workmgr-6fd5c88574-mfmpw\" (UID: \"bbd9819f-4d82-414a-9327-1abbe9d42784\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"
Sep 14 10:50:13.824378 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.824356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m56bl\" (UniqueName: \"kubernetes.io/projected/bbd9819f-4d82-414a-9327-1abbe9d42784-kube-api-access-m56bl\") pod \"klusterlet-addon-workmgr-6fd5c88574-mfmpw\" (UID: \"bbd9819f-4d82-414a-9327-1abbe9d42784\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"
Sep 14 10:50:13.895678 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.895654    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"
Sep 14 10:50:13.909987 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.909961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2sd6\" (UniqueName: \"kubernetes.io/projected/00887c1f-3e0f-4a86-8e54-35e02c9ffa53-kube-api-access-s2sd6\") pod \"downloads-76f99584f4-2rk97\" (UID: \"00887c1f-3e0f-4a86-8e54-35e02c9ffa53\") " pod="openshift-console/downloads-76f99584f4-2rk97"
Sep 14 10:50:13.921813 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.921797    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2sd6\" (UniqueName: \"kubernetes.io/projected/00887c1f-3e0f-4a86-8e54-35e02c9ffa53-kube-api-access-s2sd6\") pod \"downloads-76f99584f4-2rk97\" (UID: \"00887c1f-3e0f-4a86-8e54-35e02c9ffa53\") " pod="openshift-console/downloads-76f99584f4-2rk97"
Sep 14 10:50:13.979880 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:13.979851    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-2rk97"
Sep 14 10:50:14.167807 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:14.167735    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"]
Sep 14 10:50:14.171415 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:50:14.171388    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd9819f_4d82_414a_9327_1abbe9d42784.slice/crio-ee6d9736aab6ac92cc207cd81f5385717c907700a1270c30931d66c324292b7d WatchSource:0}: Error finding container ee6d9736aab6ac92cc207cd81f5385717c907700a1270c30931d66c324292b7d: Status 404 returned error can't find the container with id ee6d9736aab6ac92cc207cd81f5385717c907700a1270c30931d66c324292b7d
Sep 14 10:50:14.248310 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:14.248257    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-2rk97"]
Sep 14 10:50:14.250041 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:50:14.250020    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00887c1f_3e0f_4a86_8e54_35e02c9ffa53.slice/crio-49d8ac74b5c7837bdc3ac476e21baa4d2caa46ac4cb1394e0bf45d877b622a07 WatchSource:0}: Error finding container 49d8ac74b5c7837bdc3ac476e21baa4d2caa46ac4cb1394e0bf45d877b622a07: Status 404 returned error can't find the container with id 49d8ac74b5c7837bdc3ac476e21baa4d2caa46ac4cb1394e0bf45d877b622a07
Sep 14 10:50:14.282996 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:14.282962    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-2rk97" event={"ID":"00887c1f-3e0f-4a86-8e54-35e02c9ffa53","Type":"ContainerStarted","Data":"49d8ac74b5c7837bdc3ac476e21baa4d2caa46ac4cb1394e0bf45d877b622a07"}
Sep 14 10:50:14.284018 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:14.283999    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw" event={"ID":"bbd9819f-4d82-414a-9327-1abbe9d42784","Type":"ContainerStarted","Data":"ee6d9736aab6ac92cc207cd81f5385717c907700a1270c30931d66c324292b7d"}
Sep 14 10:50:19.304262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:19.304217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw" event={"ID":"bbd9819f-4d82-414a-9327-1abbe9d42784","Type":"ContainerStarted","Data":"9b12d3ba14b5cf0ce27334e0eaabb5ccc752603242fb80cd2681605334f4df22"}
Sep 14 10:50:19.304688 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:19.304387    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"
Sep 14 10:50:19.306199 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:19.306179    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw"
Sep 14 10:50:19.320641 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:19.320595    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fd5c88574-mfmpw" podStartSLOduration=1.679348769 podStartE2EDuration="6.320577865s" totalImagesPullingTime="4.641229096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:19.320265522 +0000 UTC m=+124.004949911" watchObservedRunningTime="2026-09-14 10:50:19.320577865 +0000 UTC m=+124.005262255"
Sep 14 10:50:19.464014 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:19.463969    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl"]
Sep 14 10:50:19.467332 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:19.467306    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl"
Sep 14 10:50:19.469858 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:19.469831    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 14 10:50:19.469858 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:19.469846    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vp2qv\""
Sep 14 10:50:19.476763 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:19.476701    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl"]
Sep 14 10:50:19.558109 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:19.558039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c871359e-4498-4328-977b-0ddbd9dd39a7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mr9bl\" (UID: \"c871359e-4498-4328-977b-0ddbd9dd39a7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl"
Sep 14 10:50:19.659541 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:19.659504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c871359e-4498-4328-977b-0ddbd9dd39a7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mr9bl\" (UID: \"c871359e-4498-4328-977b-0ddbd9dd39a7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl"
Sep 14 10:50:19.659703 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:19.659661    2561 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 14 10:50:19.659778 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:19.659737    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c871359e-4498-4328-977b-0ddbd9dd39a7-tls-certificates podName:c871359e-4498-4328-977b-0ddbd9dd39a7 nodeName:}" failed. No retries permitted until 2026-09-14 10:50:20.159705882 +0000 UTC m=+124.844390254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/c871359e-4498-4328-977b-0ddbd9dd39a7-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-mr9bl" (UID: "c871359e-4498-4328-977b-0ddbd9dd39a7") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 14 10:50:20.164129 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:20.164077    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c871359e-4498-4328-977b-0ddbd9dd39a7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mr9bl\" (UID: \"c871359e-4498-4328-977b-0ddbd9dd39a7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl"
Sep 14 10:50:20.166948 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:20.166920    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c871359e-4498-4328-977b-0ddbd9dd39a7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mr9bl\" (UID: \"c871359e-4498-4328-977b-0ddbd9dd39a7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl"
Sep 14 10:50:20.264807 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:20.264769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:50:20.267188 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:20.267165    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be8bbde4-d6c1-4267-bf03-c57928bc8e04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ftztr\" (UID: \"be8bbde4-d6c1-4267-bf03-c57928bc8e04\") " pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:50:20.379703 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:20.379670    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl"
Sep 14 10:50:20.510689 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:20.510656    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6n6bp\""
Sep 14 10:50:20.519037 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:20.519005    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ftztr"
Sep 14 10:50:20.657844 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:20.657811    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl"]
Sep 14 10:50:20.661193 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:50:20.661160    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc871359e_4498_4328_977b_0ddbd9dd39a7.slice/crio-26637e1cab1f243fd0aee1b9bb9dd300c9162ba24451492a24f12249b1ec8807 WatchSource:0}: Error finding container 26637e1cab1f243fd0aee1b9bb9dd300c9162ba24451492a24f12249b1ec8807: Status 404 returned error can't find the container with id 26637e1cab1f243fd0aee1b9bb9dd300c9162ba24451492a24f12249b1ec8807
Sep 14 10:50:20.794767 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:20.794672    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ftztr"]
Sep 14 10:50:20.797826 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:50:20.797782    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8bbde4_d6c1_4267_bf03_c57928bc8e04.slice/crio-a8af4d18ab25399c0fec7d85021bc13ab3653d63055c79d16bd94629a1d32bab WatchSource:0}: Error finding container a8af4d18ab25399c0fec7d85021bc13ab3653d63055c79d16bd94629a1d32bab: Status 404 returned error can't find the container with id a8af4d18ab25399c0fec7d85021bc13ab3653d63055c79d16bd94629a1d32bab
Sep 14 10:50:21.313878 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:21.313808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl" event={"ID":"c871359e-4498-4328-977b-0ddbd9dd39a7","Type":"ContainerStarted","Data":"26637e1cab1f243fd0aee1b9bb9dd300c9162ba24451492a24f12249b1ec8807"}
Sep 14 10:50:21.316598 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:21.316491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ftztr" event={"ID":"be8bbde4-d6c1-4267-bf03-c57928bc8e04","Type":"ContainerStarted","Data":"0da7abb5ed1a174386f390c8abe7e4dd12acd906e3be298d8cb8b4488790f806"}
Sep 14 10:50:21.316598 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:21.316547    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ftztr" event={"ID":"be8bbde4-d6c1-4267-bf03-c57928bc8e04","Type":"ContainerStarted","Data":"a8af4d18ab25399c0fec7d85021bc13ab3653d63055c79d16bd94629a1d32bab"}
Sep 14 10:50:22.326585 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:22.326491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ftztr" event={"ID":"be8bbde4-d6c1-4267-bf03-c57928bc8e04","Type":"ContainerStarted","Data":"0e10b0caa3c4ad27125d05af7c7d13ad7e38bd16ecd0d86d4d96f630f13ba095"}
Sep 14 10:50:22.328607 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:22.328579    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl" event={"ID":"c871359e-4498-4328-977b-0ddbd9dd39a7","Type":"ContainerStarted","Data":"1e43ab070c825d93effb05ab04cd9d3873b0986f3d9be90086545d3ec2f83c52"}
Sep 14 10:50:22.328840 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:22.328814    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl"
Sep 14 10:50:22.334476 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:22.334455    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl"
Sep 14 10:50:22.345287 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:22.345243    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mr9bl" podStartSLOduration=2.22705039 podStartE2EDuration="3.345225221s" totalImagesPullingTime="1.118174831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:22.344205246 +0000 UTC m=+127.028889636" watchObservedRunningTime="2026-09-14 10:50:22.345225221 +0000 UTC m=+127.029909613"
Sep 14 10:50:25.368060 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.368031    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d59765bdc-rs8xs"]
Sep 14 10:50:25.371492 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.371469    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.375068 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.375049    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 14 10:50:25.376197 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.376164    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 14 10:50:25.376300 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.376177    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 14 10:50:25.376300 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.376242    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 14 10:50:25.376300 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.376244    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 14 10:50:25.376480 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.376304    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-75bjs\""
Sep 14 10:50:25.382790 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.382771    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 14 10:50:25.385337 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.385316    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d59765bdc-rs8xs"]
Sep 14 10:50:25.512223 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.512189    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-config\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.512223 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.512225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-serving-cert\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.512453 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.512288    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-oauth-config\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.512453 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.512330    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7n2\" (UniqueName: \"kubernetes.io/projected/1b44ae89-526e-4b83-a81b-85e9e9deb339-kube-api-access-xl7n2\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.512453 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.512439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-service-ca\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.512608 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.512469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-trusted-ca-bundle\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.512608 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.512571    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-oauth-serving-cert\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.612960 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.612923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-service-ca\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.612960 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.612959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-trusted-ca-bundle\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.613200 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.612987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:50:25.613200 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.613024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-oauth-serving-cert\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.613200 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.613055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-config\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.613200 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.613081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-serving-cert\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.613200 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.613113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-oauth-config\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.613200 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.613144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7n2\" (UniqueName: \"kubernetes.io/projected/1b44ae89-526e-4b83-a81b-85e9e9deb339-kube-api-access-xl7n2\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.613894 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.613863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-service-ca\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.614004 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.613983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-trusted-ca-bundle\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.614065 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.613983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-config\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.614544 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.614509    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-oauth-serving-cert\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.615945 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.615920    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d78350-bf09-4796-9e47-d39ec74fde78-metrics-certs\") pod \"network-metrics-daemon-67w9c\" (UID: \"b3d78350-bf09-4796-9e47-d39ec74fde78\") " pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:50:25.616293 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.616273    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-serving-cert\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.616417 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.616398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-oauth-config\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.624267 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.624218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7n2\" (UniqueName: \"kubernetes.io/projected/1b44ae89-526e-4b83-a81b-85e9e9deb339-kube-api-access-xl7n2\") pod \"console-d59765bdc-rs8xs\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") " pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.683520 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.683488    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:25.811344 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.811319    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bckh8\""
Sep 14 10:50:25.818856 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:25.818828    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67w9c"
Sep 14 10:50:28.302287 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.302244    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"]
Sep 14 10:50:28.308468 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.308440    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hm8jf"]
Sep 14 10:50:28.308598 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.308534    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.313013 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.312694    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.315410 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.315371    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 14 10:50:28.316173 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.316131    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 14 10:50:28.316266 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.316217    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 14 10:50:28.316374 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.316359    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7tjr5\""
Sep 14 10:50:28.316636 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.316617    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 14 10:50:28.316757 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.316636    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 14 10:50:28.316883 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.316866    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-86qr8\""
Sep 14 10:50:28.316977 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.316888    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 14 10:50:28.317702 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.317684    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 14 10:50:28.318166 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.318149    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 14 10:50:28.318411 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.318396    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 14 10:50:28.337413 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.337352    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"]
Sep 14 10:50:28.340780 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.340756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db5b736c-6692-4802-9be1-2e6812fb2cd1-sys\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.340891 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.340790    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db5b736c-6692-4802-9be1-2e6812fb2cd1-metrics-client-ca\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.340891 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.340834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db5b736c-6692-4802-9be1-2e6812fb2cd1-root\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.340891 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.340859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/35e5e4ed-da13-4019-9a71-045bff257926-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.340891 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.340883    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/35e5e4ed-da13-4019-9a71-045bff257926-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.341163 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.340911    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-wtmp\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.341163 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.340996    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.341163 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.341066    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35e5e4ed-da13-4019-9a71-045bff257926-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.341163 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.341095    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-tls\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.341163 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.341128    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/35e5e4ed-da13-4019-9a71-045bff257926-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.341163 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.341161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-textfile\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.341436 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.341195    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-accelerators-collector-config\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.341436 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.341234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krg9n\" (UniqueName: \"kubernetes.io/projected/db5b736c-6692-4802-9be1-2e6812fb2cd1-kube-api-access-krg9n\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.341436 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.341286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35e5e4ed-da13-4019-9a71-045bff257926-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.341436 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.341346    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jph5\" (UniqueName: \"kubernetes.io/projected/35e5e4ed-da13-4019-9a71-045bff257926-kube-api-access-9jph5\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.442392 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442350    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jph5\" (UniqueName: \"kubernetes.io/projected/35e5e4ed-da13-4019-9a71-045bff257926-kube-api-access-9jph5\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.442562 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db5b736c-6692-4802-9be1-2e6812fb2cd1-sys\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.442562 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db5b736c-6692-4802-9be1-2e6812fb2cd1-metrics-client-ca\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.442562 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442480    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db5b736c-6692-4802-9be1-2e6812fb2cd1-root\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.442562 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442539    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db5b736c-6692-4802-9be1-2e6812fb2cd1-root\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.442758 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442582    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db5b736c-6692-4802-9be1-2e6812fb2cd1-sys\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.442758 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442605    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/35e5e4ed-da13-4019-9a71-045bff257926-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.442758 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/35e5e4ed-da13-4019-9a71-045bff257926-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.442758 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-wtmp\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.442758 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442697    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.442758 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35e5e4ed-da13-4019-9a71-045bff257926-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.443041 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-tls\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.443041 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/35e5e4ed-da13-4019-9a71-045bff257926-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.443041 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442837    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-textfile\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.443041 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-wtmp\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.443041 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-accelerators-collector-config\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.443041 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442892    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krg9n\" (UniqueName: \"kubernetes.io/projected/db5b736c-6692-4802-9be1-2e6812fb2cd1-kube-api-access-krg9n\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.443041 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442928    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35e5e4ed-da13-4019-9a71-045bff257926-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.443041 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.442943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/35e5e4ed-da13-4019-9a71-045bff257926-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.443041 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:28.442966    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 14 10:50:28.443041 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:28.443022    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-tls podName:db5b736c-6692-4802-9be1-2e6812fb2cd1 nodeName:}" failed. No retries permitted until 2026-09-14 10:50:28.94300323 +0000 UTC m=+133.627687597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-tls") pod "node-exporter-hm8jf" (UID: "db5b736c-6692-4802-9be1-2e6812fb2cd1") : secret "node-exporter-tls" not found
Sep 14 10:50:28.443506 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.443155    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db5b736c-6692-4802-9be1-2e6812fb2cd1-metrics-client-ca\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.443506 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.443219    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-textfile\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.443682 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.443635    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/35e5e4ed-da13-4019-9a71-045bff257926-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.443821 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.443731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-accelerators-collector-config\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.443888 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.443822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35e5e4ed-da13-4019-9a71-045bff257926-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.445833 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.445801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.445954 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.445905    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/35e5e4ed-da13-4019-9a71-045bff257926-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.446035 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.446011    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35e5e4ed-da13-4019-9a71-045bff257926-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.458634 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.458593    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jph5\" (UniqueName: \"kubernetes.io/projected/35e5e4ed-da13-4019-9a71-045bff257926-kube-api-access-9jph5\") pod \"kube-state-metrics-59fbdd77d-vjsgs\" (UID: \"35e5e4ed-da13-4019-9a71-045bff257926\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.489315 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.489289    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krg9n\" (UniqueName: \"kubernetes.io/projected/db5b736c-6692-4802-9be1-2e6812fb2cd1-kube-api-access-krg9n\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.623654 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.623627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"
Sep 14 10:50:28.947265 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.947183    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-tls\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:28.949766 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:28.949735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db5b736c-6692-4802-9be1-2e6812fb2cd1-node-exporter-tls\") pod \"node-exporter-hm8jf\" (UID: \"db5b736c-6692-4802-9be1-2e6812fb2cd1\") " pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:29.231149 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.231075    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hm8jf"
Sep 14 10:50:29.569871 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.569796    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:50:29.574099 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.574077    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.582845 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:29.582694    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-dockercfg-84qd7\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-84qd7\"" type="*v1.Secret"
Sep 14 10:50:29.582845 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:29.582743    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-cluster-tls-config\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret"
Sep 14 10:50:29.582845 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:29.582733    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-tls-assets-0\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret"
Sep 14 10:50:29.582845 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:29.582733    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-generated\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret"
Sep 14 10:50:29.582845 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:29.582727    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-web-config\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" type="*v1.Secret"
Sep 14 10:50:29.582845 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:29.582780    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"alertmanager-main-0\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.582845 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:29.582793    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-tls\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret"
Sep 14 10:50:29.582845 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:29.582809    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret"
Sep 14 10:50:29.582845 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:29.582817    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"alertmanager-trusted-ca-bundle\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" type="*v1.ConfigMap"
Sep 14 10:50:29.583297 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:29.582856    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-kube-rbac-proxy-metric\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret"
Sep 14 10:50:29.583297 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:29.582921    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-kube-rbac-proxy-web\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" type="*v1.Secret"
Sep 14 10:50:29.617570 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.617529    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:50:29.654859 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.654834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-web-config\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.654982 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.654882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.654982 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.654909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.654982 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.654962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.655200 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.655013    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.655200 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.655069    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.655200 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.655113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-out\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.655200 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.655146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.655200 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.655187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dql44\" (UniqueName: \"kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-kube-api-access-dql44\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.655410 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.655249    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.655410 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.655283    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.655410 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.655309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.655410 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.655357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.756239 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.756203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dql44\" (UniqueName: \"kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-kube-api-access-dql44\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.756404 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.756346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.756404 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.756377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.756404 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.756395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.756561 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.756419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.756561 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.756451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-web-config\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.756650 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.756555    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.756650 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.756606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.756650 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.756635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.756818 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.756677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.756818 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.756735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.756818 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.756768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-out\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.756818 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.756801    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.757158 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.757097    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.757549 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.757525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.759314 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.759291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-out\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:29.785223 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:29.785200    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dql44\" (UniqueName: \"kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-kube-api-access-dql44\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:30.431887 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.431844    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-84qd7\""
Sep 14 10:50:30.507890 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.507858    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 14 10:50:30.517845 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.517807    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:30.532374 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.532344    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 14 10:50:30.539370 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.539347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:30.620455 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.620425    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 14 10:50:30.629333 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.629307    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-web-config\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:30.727590 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.727516    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 14 10:50:30.739396 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.739370    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:30.757353 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:30.757327    2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-cluster-tls-config: failed to sync secret cache: timed out waiting for the condition
Sep 14 10:50:30.757353 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:30.757348    2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-kube-rbac-proxy-metric: failed to sync secret cache: timed out waiting for the condition
Sep 14 10:50:30.757539 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:30.757352    2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-generated: failed to sync secret cache: timed out waiting for the condition
Sep 14 10:50:30.757539 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:30.757324    2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-kube-rbac-proxy: failed to sync secret cache: timed out waiting for the condition
Sep 14 10:50:30.757539 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:30.757352    2561 projected.go:266] Couldn't get secret openshift-monitoring/alertmanager-main-tls-assets-0: failed to sync secret cache: timed out waiting for the condition
Sep 14 10:50:30.757539 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:30.757448    2561 projected.go:196] Error preparing data for projected volume tls-assets for pod openshift-monitoring/alertmanager-main-0: failed to sync secret cache: timed out waiting for the condition
Sep 14 10:50:30.757539 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:30.757410    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-cluster-tls-config podName:7a2f546a-c63a-4c1b-af96-84dd7b25bc3d nodeName:}" failed. No retries permitted until 2026-09-14 10:50:31.257391637 +0000 UTC m=+135.942076005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-tls-config" (UniqueName: "kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-cluster-tls-config") pod "alertmanager-main-0" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d") : failed to sync secret cache: timed out waiting for the condition
Sep 14 10:50:30.757539 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:30.757514    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-metric podName:7a2f546a-c63a-4c1b-af96-84dd7b25bc3d nodeName:}" failed. No retries permitted until 2026-09-14 10:50:31.257488509 +0000 UTC m=+135.942172889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-kube-rbac-proxy-metric" (UniqueName: "kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-metric") pod "alertmanager-main-0" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d") : failed to sync secret cache: timed out waiting for the condition
Sep 14 10:50:30.757539 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:30.757533    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-volume podName:7a2f546a-c63a-4c1b-af96-84dd7b25bc3d nodeName:}" failed. No retries permitted until 2026-09-14 10:50:31.257522628 +0000 UTC m=+135.942207001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config-volume" (UniqueName: "kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-volume") pod "alertmanager-main-0" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d") : failed to sync secret cache: timed out waiting for the condition
Sep 14 10:50:30.757831 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:30.757565    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy podName:7a2f546a-c63a-4c1b-af96-84dd7b25bc3d nodeName:}" failed. No retries permitted until 2026-09-14 10:50:31.257540438 +0000 UTC m=+135.942224811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-kube-rbac-proxy" (UniqueName: "kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy") pod "alertmanager-main-0" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d") : failed to sync secret cache: timed out waiting for the condition
Sep 14 10:50:30.757831 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:30.757580    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-tls-assets podName:7a2f546a-c63a-4c1b-af96-84dd7b25bc3d nodeName:}" failed. No retries permitted until 2026-09-14 10:50:31.257571848 +0000 UTC m=+135.942256220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-assets" (UniqueName: "kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-tls-assets") pod "alertmanager-main-0" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d") : failed to sync secret cache: timed out waiting for the condition
Sep 14 10:50:30.764834 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.764798    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 14 10:50:30.771916 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.771896    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 14 10:50:30.830960 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.830936    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 14 10:50:30.844040 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:50:30.843915    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5b736c_6692_4802_9be1_2e6812fb2cd1.slice/crio-8b79aa49f8a7541544cd51cfa647153410f661dde62456e9eece7caf43a7bea7 WatchSource:0}: Error finding container 8b79aa49f8a7541544cd51cfa647153410f661dde62456e9eece7caf43a7bea7: Status 404 returned error can't find the container with id 8b79aa49f8a7541544cd51cfa647153410f661dde62456e9eece7caf43a7bea7
Sep 14 10:50:30.856000 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.854319    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 14 10:50:30.970205 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:30.970110    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 14 10:50:31.185651 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.185515    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs"]
Sep 14 10:50:31.185939 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.185786    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-67w9c"]
Sep 14 10:50:31.190661 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:50:31.190619    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b44ae89_526e_4b83_a81b_85e9e9deb339.slice/crio-eba50d51abacfc0f2efff6ef801767d4604f7c0399f5a1fb17d83f707defebb1 WatchSource:0}: Error finding container eba50d51abacfc0f2efff6ef801767d4604f7c0399f5a1fb17d83f707defebb1: Status 404 returned error can't find the container with id eba50d51abacfc0f2efff6ef801767d4604f7c0399f5a1fb17d83f707defebb1
Sep 14 10:50:31.190968 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.190937    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d59765bdc-rs8xs"]
Sep 14 10:50:31.272180 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.272106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:31.272180 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.272158    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:31.272417 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.272220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:31.272723 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.272661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:31.272849 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.272782    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:31.275283 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.275259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:31.275620 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.275579    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:31.275756 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.275736    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:31.275830 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.275809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:31.276168 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.276148    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:31.365922 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.365859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d59765bdc-rs8xs" event={"ID":"1b44ae89-526e-4b83-a81b-85e9e9deb339","Type":"ContainerStarted","Data":"eba50d51abacfc0f2efff6ef801767d4604f7c0399f5a1fb17d83f707defebb1"}
Sep 14 10:50:31.367410 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.367370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-67w9c" event={"ID":"b3d78350-bf09-4796-9e47-d39ec74fde78","Type":"ContainerStarted","Data":"bdf3913f055ac10c05e02c380a11a52c068b045a72f8941514f3a9074d022c20"}
Sep 14 10:50:31.370124 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.370048    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ftztr" event={"ID":"be8bbde4-d6c1-4267-bf03-c57928bc8e04","Type":"ContainerStarted","Data":"7d17a1b29087edadb4284b3e1ea792d03869df1f0049b8cb04d12ead93f30c89"}
Sep 14 10:50:31.371647 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.371598    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hm8jf" event={"ID":"db5b736c-6692-4802-9be1-2e6812fb2cd1","Type":"ContainerStarted","Data":"8b79aa49f8a7541544cd51cfa647153410f661dde62456e9eece7caf43a7bea7"}
Sep 14 10:50:31.373075 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.373045    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs" event={"ID":"35e5e4ed-da13-4019-9a71-045bff257926","Type":"ContainerStarted","Data":"80e1b49b1bc9c0e07f4b2c941e7781b9eb7c646f425796e601731b901de81a75"}
Sep 14 10:50:31.374769 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.374741    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-2rk97" event={"ID":"00887c1f-3e0f-4a86-8e54-35e02c9ffa53","Type":"ContainerStarted","Data":"62b6c29820a31b17b2d65360a86744c4fe8f4e05973d6cbf1fbd4f4cb5c9d307"}
Sep 14 10:50:31.375073 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.374969    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-2rk97"
Sep 14 10:50:31.385445 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.385424    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:50:31.386835 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.386810    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-2rk97"
Sep 14 10:50:31.402558 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.402498    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ftztr" podStartSLOduration=33.512639528 podStartE2EDuration="43.402479107s" totalImagesPullingTime="9.889839579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:49:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:31.401989863 +0000 UTC m=+136.086674265" watchObservedRunningTime="2026-09-14 10:50:31.402479107 +0000 UTC m=+136.087163512"
Sep 14 10:50:31.745793 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.744846    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-2rk97" podStartSLOduration=2.086718175 podStartE2EDuration="18.744814413s" totalImagesPullingTime="16.658096238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:31.509181412 +0000 UTC m=+136.193865800" watchObservedRunningTime="2026-09-14 10:50:31.744814413 +0000 UTC m=+136.429498805"
Sep 14 10:50:31.747947 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:31.747212    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:50:32.385297 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:32.384783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerStarted","Data":"1f77af44a9bf2ea788b8050dd5c39f8d05787a629588f94a492f8ebe0f7e46f8"}
Sep 14 10:50:33.039475 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:33.039439    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d"]
Sep 14 10:50:33.068264 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:33.065183    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d"
Sep 14 10:50:33.079389 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:33.079359    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dmz4m\""
Sep 14 10:50:33.079664 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:33.079637    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 14 10:50:33.084670 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:33.084643    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d"]
Sep 14 10:50:33.097287 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:33.097257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmj9d\" (UniqueName: \"kubernetes.io/projected/0a8b5c7b-6176-47f7-b7f8-764efbf6f428-kube-api-access-xmj9d\") pod \"monitoring-plugin-5d85778b9d-5vz8d\" (UID: \"0a8b5c7b-6176-47f7-b7f8-764efbf6f428\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d"
Sep 14 10:50:33.097408 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:33.097374    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a8b5c7b-6176-47f7-b7f8-764efbf6f428-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5vz8d\" (UID: \"0a8b5c7b-6176-47f7-b7f8-764efbf6f428\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d"
Sep 14 10:50:33.198466 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:33.198421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a8b5c7b-6176-47f7-b7f8-764efbf6f428-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5vz8d\" (UID: \"0a8b5c7b-6176-47f7-b7f8-764efbf6f428\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d"
Sep 14 10:50:33.198661 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:33.198500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmj9d\" (UniqueName: \"kubernetes.io/projected/0a8b5c7b-6176-47f7-b7f8-764efbf6f428-kube-api-access-xmj9d\") pod \"monitoring-plugin-5d85778b9d-5vz8d\" (UID: \"0a8b5c7b-6176-47f7-b7f8-764efbf6f428\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d"
Sep 14 10:50:33.202875 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:33.202259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a8b5c7b-6176-47f7-b7f8-764efbf6f428-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5vz8d\" (UID: \"0a8b5c7b-6176-47f7-b7f8-764efbf6f428\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d"
Sep 14 10:50:33.241580 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:33.241538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmj9d\" (UniqueName: \"kubernetes.io/projected/0a8b5c7b-6176-47f7-b7f8-764efbf6f428-kube-api-access-xmj9d\") pod \"monitoring-plugin-5d85778b9d-5vz8d\" (UID: \"0a8b5c7b-6176-47f7-b7f8-764efbf6f428\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d"
Sep 14 10:50:33.385595 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:33.385566    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d"
Sep 14 10:50:34.086704 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:34.086671    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d59765bdc-rs8xs"]
Sep 14 10:50:35.392382 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.392347    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-694df55669-jswp6"]
Sep 14 10:50:35.393184 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:35.392687    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-694df55669-jswp6" podUID="396c0cfd-285d-4d44-9260-ac4c7d3fffb7"
Sep 14 10:50:35.399860 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.399835    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:50:35.459600 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.459567    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:50:35.522819 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.522785    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-image-registry-private-configuration\") pod \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") "
Sep 14 10:50:35.523082 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.522847    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-ca-trust-extracted\") pod \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") "
Sep 14 10:50:35.523082 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.522882    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-installation-pull-secrets\") pod \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") "
Sep 14 10:50:35.523082 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.522925    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-trusted-ca\") pod \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") "
Sep 14 10:50:35.523690 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.523181    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "396c0cfd-285d-4d44-9260-ac4c7d3fffb7" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:35.523690 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.523204    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-certificates\") pod \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") "
Sep 14 10:50:35.523690 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.523274    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-bound-sa-token\") pod \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") "
Sep 14 10:50:35.523690 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.523313    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftsqv\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-kube-api-access-ftsqv\") pod \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\" (UID: \"396c0cfd-285d-4d44-9260-ac4c7d3fffb7\") "
Sep 14 10:50:35.523690 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.523315    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "396c0cfd-285d-4d44-9260-ac4c7d3fffb7" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:50:35.523690 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.523603    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-ca-trust-extracted\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:50:35.523690 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.523623    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-trusted-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:50:35.524404 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.524004    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "396c0cfd-285d-4d44-9260-ac4c7d3fffb7" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:50:35.527326 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.527284    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "396c0cfd-285d-4d44-9260-ac4c7d3fffb7" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:50:35.527855 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.527779    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "396c0cfd-285d-4d44-9260-ac4c7d3fffb7" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:50:35.528285 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.528243    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-kube-api-access-ftsqv" (OuterVolumeSpecName: "kube-api-access-ftsqv") pod "396c0cfd-285d-4d44-9260-ac4c7d3fffb7" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7"). InnerVolumeSpecName "kube-api-access-ftsqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:50:35.528601 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.528574    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "396c0cfd-285d-4d44-9260-ac4c7d3fffb7" (UID: "396c0cfd-285d-4d44-9260-ac4c7d3fffb7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:50:35.624588 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.624552    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-image-registry-private-configuration\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:50:35.624588 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.624586    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-installation-pull-secrets\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:50:35.624852 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.624601    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-certificates\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:50:35.624852 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.624615    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-bound-sa-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:50:35.624852 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:35.624629    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftsqv\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-kube-api-access-ftsqv\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:50:36.403978 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:36.403946    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-694df55669-jswp6"
Sep 14 10:50:36.448219 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:36.448183    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-694df55669-jswp6"]
Sep 14 10:50:36.449961 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:36.449930    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-694df55669-jswp6"]
Sep 14 10:50:36.533549 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:36.533128    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/396c0cfd-285d-4d44-9260-ac4c7d3fffb7-registry-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:50:36.765840 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:36.765551    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d"]
Sep 14 10:50:36.775497 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:50:36.775472    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8b5c7b_6176_47f7_b7f8_764efbf6f428.slice/crio-1a54f9b32334790441e9bb6363fa8f19835f7d70865f95f5ec30d90a2660a78f WatchSource:0}: Error finding container 1a54f9b32334790441e9bb6363fa8f19835f7d70865f95f5ec30d90a2660a78f: Status 404 returned error can't find the container with id 1a54f9b32334790441e9bb6363fa8f19835f7d70865f95f5ec30d90a2660a78f
Sep 14 10:50:37.409787 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.409745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d59765bdc-rs8xs" event={"ID":"1b44ae89-526e-4b83-a81b-85e9e9deb339","Type":"ContainerStarted","Data":"86e432cbc6e66c6454fcb362a20f636e97d7a686f725f886c9ded0aa59ae0dee"}
Sep 14 10:50:37.412141 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.412113    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-67w9c" event={"ID":"b3d78350-bf09-4796-9e47-d39ec74fde78","Type":"ContainerStarted","Data":"48d3ca6a864f0e50eb3b47e425cfc86367b455473d19d14e3c3763dd1d123acd"}
Sep 14 10:50:37.412334 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.412314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-67w9c" event={"ID":"b3d78350-bf09-4796-9e47-d39ec74fde78","Type":"ContainerStarted","Data":"3a86d5b963e1b2fcd2a8f99d1f5d3e0680556a8985fd34271125152fa13e0098"}
Sep 14 10:50:37.414383 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.414356    2561 generic.go:356] "Generic (PLEG): container finished" podID="db5b736c-6692-4802-9be1-2e6812fb2cd1" containerID="aacb894ae199f5e81883656ebbe6675a7e9f17c70d52e62175ed9a4bb1b53600" exitCode=0
Sep 14 10:50:37.414519 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.414432    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hm8jf" event={"ID":"db5b736c-6692-4802-9be1-2e6812fb2cd1","Type":"ContainerDied","Data":"aacb894ae199f5e81883656ebbe6675a7e9f17c70d52e62175ed9a4bb1b53600"}
Sep 14 10:50:37.415874 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.415851    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d" event={"ID":"0a8b5c7b-6176-47f7-b7f8-764efbf6f428","Type":"ContainerStarted","Data":"1a54f9b32334790441e9bb6363fa8f19835f7d70865f95f5ec30d90a2660a78f"}
Sep 14 10:50:37.422306 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.422272    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs" event={"ID":"35e5e4ed-da13-4019-9a71-045bff257926","Type":"ContainerStarted","Data":"ee29a1c56844796c919060bf09cdc5535e5e2a5ded2b5212027f6c711c937203"}
Sep 14 10:50:37.422404 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.422344    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs" event={"ID":"35e5e4ed-da13-4019-9a71-045bff257926","Type":"ContainerStarted","Data":"5dd39a9468a6e204b04d27c15f9e44be575f707f7a429ddfb7669cb0f8d02afc"}
Sep 14 10:50:37.422404 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.422375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs" event={"ID":"35e5e4ed-da13-4019-9a71-045bff257926","Type":"ContainerStarted","Data":"f8437a1f02458bc89713541dea9f827e1112a8be6571271ed634e97026429559"}
Sep 14 10:50:37.424081 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.424057    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerID="b5a9c76037632622ef089ec8af41d55fff09ab73822c86c19ff10c99ad2bed8a" exitCode=0
Sep 14 10:50:37.424186 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.424127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerDied","Data":"b5a9c76037632622ef089ec8af41d55fff09ab73822c86c19ff10c99ad2bed8a"}
Sep 14 10:50:37.429809 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.429765    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d59765bdc-rs8xs" podStartSLOduration=6.938579207 podStartE2EDuration="12.429752764s" totalImagesPullingTime="5.491173557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:37.428820441 +0000 UTC m=+142.113504833" watchObservedRunningTime="2026-09-14 10:50:37.429752764 +0000 UTC m=+142.114437168"
Sep 14 10:50:37.470610 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.470553    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vjsgs" podStartSLOduration=4.000982714 podStartE2EDuration="9.470535676s" totalImagesPullingTime="5.469552962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:37.469515196 +0000 UTC m=+142.154199586" watchObservedRunningTime="2026-09-14 10:50:37.470535676 +0000 UTC m=+142.155220067"
Sep 14 10:50:37.525065 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.525011    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-67w9c" podStartSLOduration=137.037472726 podStartE2EDuration="2m22.524989092s" totalImagesPullingTime="5.487516366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:37.523080209 +0000 UTC m=+142.207764601" watchObservedRunningTime="2026-09-14 10:50:37.524989092 +0000 UTC m=+142.209673481"
Sep 14 10:50:37.804390 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:37.804347    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="396c0cfd-285d-4d44-9260-ac4c7d3fffb7" path="/var/lib/kubelet/pods/396c0cfd-285d-4d44-9260-ac4c7d3fffb7/volumes"
Sep 14 10:50:38.434982 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:38.434859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hm8jf" event={"ID":"db5b736c-6692-4802-9be1-2e6812fb2cd1","Type":"ContainerStarted","Data":"6c63244094bfd087f015d77c8d6220dd07ec99f887f7971063d725bab73fde56"}
Sep 14 10:50:38.439458 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:38.436317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hm8jf" event={"ID":"db5b736c-6692-4802-9be1-2e6812fb2cd1","Type":"ContainerStarted","Data":"c31f3a48c57cad504035b9ec0d3b67242460d792d6ab721b0800a17740132619"}
Sep 14 10:50:38.470053 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:38.469476    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hm8jf" podStartSLOduration=4.482299572 podStartE2EDuration="10.469454113s" totalImagesPullingTime="5.818939242s" totalInitContainerRuntime="168.215299ms" isStatefulPod=true podCreationTimestamp="2026-09-14 10:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:38.466474005 +0000 UTC m=+143.151158393" watchObservedRunningTime="2026-09-14 10:50:38.469454113 +0000 UTC m=+143.154138503"
Sep 14 10:50:40.448290 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:40.448255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerStarted","Data":"8d620baf3f4d9331a08934f9626da568605678fdcf52fee9211532bfd77275df"}
Sep 14 10:50:40.448633 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:40.448343    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerStarted","Data":"da681e6fdd5ce46924682c7c5c0b10239a86ffe52d08d89c406650bec91b2640"}
Sep 14 10:50:40.448633 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:40.448389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerStarted","Data":"255595d8621498dec8db6f26af9b7945d5ab2542514bd1c36e014062dd220517"}
Sep 14 10:50:40.448633 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:40.448427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerStarted","Data":"d6fa3257d6f0b2ea7ce2816a8cec67ceec6c7bfdfcb6121c9c844cb43e0078a1"}
Sep 14 10:50:40.448633 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:40.448465    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerStarted","Data":"7896b2353217ea5dfb49f3f1e08911c0d465f0c267b1e2e4644b60d809283b8b"}
Sep 14 10:50:40.449655 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:40.449625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d" event={"ID":"0a8b5c7b-6176-47f7-b7f8-764efbf6f428","Type":"ContainerStarted","Data":"702a19aa46b9f7d86bae6da6e948cc4a7fcd0025ad526ad28f45eaead0b05ad1"}
Sep 14 10:50:40.449953 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:40.449862    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d"
Sep 14 10:50:40.454403 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:40.454382    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d"
Sep 14 10:50:40.518969 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:40.518929    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5vz8d" podStartSLOduration=5.745982587 podStartE2EDuration="8.518915208s" totalImagesPullingTime="2.772932621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:40.480593873 +0000 UTC m=+145.165278263" watchObservedRunningTime="2026-09-14 10:50:40.518915208 +0000 UTC m=+145.203599596"
Sep 14 10:50:42.461474 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:42.461433    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerStarted","Data":"688697e169da9b7c6dadac23c442c6efa162c44063e2a328c5ef2d3883149b83"}
Sep 14 10:50:42.503201 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:42.503156    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.599702393 podStartE2EDuration="13.503141239s" totalImagesPullingTime="8.866447784s" totalInitContainerRuntime="36.991062ms" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:42.500733223 +0000 UTC m=+147.185417609" watchObservedRunningTime="2026-09-14 10:50:42.503141239 +0000 UTC m=+147.187825627"
Sep 14 10:50:45.684247 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:45.684215    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:50:52.694239 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:50:52.694194    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-lbplh" podUID="90f711db-6849-4eb1-9ed2-85b410b8765e"
Sep 14 10:50:53.502423 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:53.502392    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lbplh"
Sep 14 10:50:57.614572 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:57.614537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:50:57.616882 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:57.616861    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90f711db-6849-4eb1-9ed2-85b410b8765e-metrics-tls\") pod \"dns-default-lbplh\" (UID: \"90f711db-6849-4eb1-9ed2-85b410b8765e\") " pod="openshift-dns/dns-default-lbplh"
Sep 14 10:50:57.706445 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:57.706421    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qfwcj\""
Sep 14 10:50:57.713805 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:57.713789    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lbplh"
Sep 14 10:50:58.012951 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:58.012927    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lbplh"]
Sep 14 10:50:58.015247 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:50:58.015219    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f711db_6849_4eb1_9ed2_85b410b8765e.slice/crio-b9f5ff137271fbc9b0911f761b0ec0ce8aecdda8b2a97e33918e15a915d52007 WatchSource:0}: Error finding container b9f5ff137271fbc9b0911f761b0ec0ce8aecdda8b2a97e33918e15a915d52007: Status 404 returned error can't find the container with id b9f5ff137271fbc9b0911f761b0ec0ce8aecdda8b2a97e33918e15a915d52007
Sep 14 10:50:58.521939 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:50:58.521896    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lbplh" event={"ID":"90f711db-6849-4eb1-9ed2-85b410b8765e","Type":"ContainerStarted","Data":"b9f5ff137271fbc9b0911f761b0ec0ce8aecdda8b2a97e33918e15a915d52007"}
Sep 14 10:51:00.531447 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:00.531409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lbplh" event={"ID":"90f711db-6849-4eb1-9ed2-85b410b8765e","Type":"ContainerStarted","Data":"8bbaf1e998b5e0ad37062e3bc8964c6220e18abe5f248761bece771aab1113d7"}
Sep 14 10:51:00.531817 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:00.531469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lbplh" event={"ID":"90f711db-6849-4eb1-9ed2-85b410b8765e","Type":"ContainerStarted","Data":"a0461cd7f4a0e8ab167037303d120e1cd0630aa173f9fd29f07d9be4e288f594"}
Sep 14 10:51:00.531817 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:00.531564    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lbplh"
Sep 14 10:51:00.569370 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:00.569325    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lbplh" podStartSLOduration=129.91805518500001 podStartE2EDuration="2m11.569311624s" totalImagesPullingTime="1.651256439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:51:00.567686702 +0000 UTC m=+165.252371091" watchObservedRunningTime="2026-09-14 10:51:00.569311624 +0000 UTC m=+165.253996050"
Sep 14 10:51:02.435536 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.435489    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d59765bdc-rs8xs" podUID="1b44ae89-526e-4b83-a81b-85e9e9deb339" containerName="console" containerID="cri-o://86e432cbc6e66c6454fcb362a20f636e97d7a686f725f886c9ded0aa59ae0dee" gracePeriod=15
Sep 14 10:51:02.706029 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.705973    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:51:02.855418 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.855389    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-config\") pod \"1b44ae89-526e-4b83-a81b-85e9e9deb339\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") "
Sep 14 10:51:02.855585 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.855430    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-service-ca\") pod \"1b44ae89-526e-4b83-a81b-85e9e9deb339\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") "
Sep 14 10:51:02.855585 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.855459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-trusted-ca-bundle\") pod \"1b44ae89-526e-4b83-a81b-85e9e9deb339\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") "
Sep 14 10:51:02.855585 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.855487    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-serving-cert\") pod \"1b44ae89-526e-4b83-a81b-85e9e9deb339\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") "
Sep 14 10:51:02.855585 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.855517    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-oauth-config\") pod \"1b44ae89-526e-4b83-a81b-85e9e9deb339\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") "
Sep 14 10:51:02.855585 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.855560    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl7n2\" (UniqueName: \"kubernetes.io/projected/1b44ae89-526e-4b83-a81b-85e9e9deb339-kube-api-access-xl7n2\") pod \"1b44ae89-526e-4b83-a81b-85e9e9deb339\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") "
Sep 14 10:51:02.855858 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.855622    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-oauth-serving-cert\") pod \"1b44ae89-526e-4b83-a81b-85e9e9deb339\" (UID: \"1b44ae89-526e-4b83-a81b-85e9e9deb339\") "
Sep 14 10:51:02.855964 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.855934    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-config" (OuterVolumeSpecName: "console-config") pod "1b44ae89-526e-4b83-a81b-85e9e9deb339" (UID: "1b44ae89-526e-4b83-a81b-85e9e9deb339"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:51:02.855964 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.855952    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-service-ca" (OuterVolumeSpecName: "service-ca") pod "1b44ae89-526e-4b83-a81b-85e9e9deb339" (UID: "1b44ae89-526e-4b83-a81b-85e9e9deb339"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:51:02.856108 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.855972    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1b44ae89-526e-4b83-a81b-85e9e9deb339" (UID: "1b44ae89-526e-4b83-a81b-85e9e9deb339"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:51:02.856185 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.856157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1b44ae89-526e-4b83-a81b-85e9e9deb339" (UID: "1b44ae89-526e-4b83-a81b-85e9e9deb339"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:51:02.857952 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.857923    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1b44ae89-526e-4b83-a81b-85e9e9deb339" (UID: "1b44ae89-526e-4b83-a81b-85e9e9deb339"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:51:02.858030 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.857961    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b44ae89-526e-4b83-a81b-85e9e9deb339-kube-api-access-xl7n2" (OuterVolumeSpecName: "kube-api-access-xl7n2") pod "1b44ae89-526e-4b83-a81b-85e9e9deb339" (UID: "1b44ae89-526e-4b83-a81b-85e9e9deb339"). InnerVolumeSpecName "kube-api-access-xl7n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:51:02.858079 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.858024    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1b44ae89-526e-4b83-a81b-85e9e9deb339" (UID: "1b44ae89-526e-4b83-a81b-85e9e9deb339"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:51:02.956858 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.956800    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:02.956858 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.956820    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-trusted-ca-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:02.956858 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.956830    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-serving-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:02.956858 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.956839    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-oauth-config\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:02.956858 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.956847    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl7n2\" (UniqueName: \"kubernetes.io/projected/1b44ae89-526e-4b83-a81b-85e9e9deb339-kube-api-access-xl7n2\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:02.956858 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.956855    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-oauth-serving-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:02.957103 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:02.956863    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b44ae89-526e-4b83-a81b-85e9e9deb339-console-config\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:03.545407 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:03.545373    2561 generic.go:356] "Generic (PLEG): container finished" podID="1b44ae89-526e-4b83-a81b-85e9e9deb339" containerID="86e432cbc6e66c6454fcb362a20f636e97d7a686f725f886c9ded0aa59ae0dee" exitCode=0
Sep 14 10:51:03.545801 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:03.545426    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d59765bdc-rs8xs"
Sep 14 10:51:03.545801 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:03.545460    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d59765bdc-rs8xs" event={"ID":"1b44ae89-526e-4b83-a81b-85e9e9deb339","Type":"ContainerDied","Data":"86e432cbc6e66c6454fcb362a20f636e97d7a686f725f886c9ded0aa59ae0dee"}
Sep 14 10:51:03.545801 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:03.545510    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d59765bdc-rs8xs" event={"ID":"1b44ae89-526e-4b83-a81b-85e9e9deb339","Type":"ContainerDied","Data":"eba50d51abacfc0f2efff6ef801767d4604f7c0399f5a1fb17d83f707defebb1"}
Sep 14 10:51:03.545801 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:03.545533    2561 scope.go:117] "RemoveContainer" containerID="86e432cbc6e66c6454fcb362a20f636e97d7a686f725f886c9ded0aa59ae0dee"
Sep 14 10:51:03.553929 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:03.553916    2561 scope.go:117] "RemoveContainer" containerID="86e432cbc6e66c6454fcb362a20f636e97d7a686f725f886c9ded0aa59ae0dee"
Sep 14 10:51:03.554183 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:51:03.554163    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e432cbc6e66c6454fcb362a20f636e97d7a686f725f886c9ded0aa59ae0dee\": container with ID starting with 86e432cbc6e66c6454fcb362a20f636e97d7a686f725f886c9ded0aa59ae0dee not found: ID does not exist" containerID="86e432cbc6e66c6454fcb362a20f636e97d7a686f725f886c9ded0aa59ae0dee"
Sep 14 10:51:03.554220 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:03.554191    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e432cbc6e66c6454fcb362a20f636e97d7a686f725f886c9ded0aa59ae0dee"} err="failed to get container status \"86e432cbc6e66c6454fcb362a20f636e97d7a686f725f886c9ded0aa59ae0dee\": rpc error: code = NotFound desc = could not find container \"86e432cbc6e66c6454fcb362a20f636e97d7a686f725f886c9ded0aa59ae0dee\": container with ID starting with 86e432cbc6e66c6454fcb362a20f636e97d7a686f725f886c9ded0aa59ae0dee not found: ID does not exist"
Sep 14 10:51:03.570084 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:03.570061    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d59765bdc-rs8xs"]
Sep 14 10:51:03.574468 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:03.574447    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d59765bdc-rs8xs"]
Sep 14 10:51:03.803204 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:03.802653    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b44ae89-526e-4b83-a81b-85e9e9deb339" path="/var/lib/kubelet/pods/1b44ae89-526e-4b83-a81b-85e9e9deb339/volumes"
Sep 14 10:51:10.537481 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:10.537407    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lbplh"
Sep 14 10:51:49.342918 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.342887    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:51:49.343457 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.343167    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="alertmanager" containerID="cri-o://7896b2353217ea5dfb49f3f1e08911c0d465f0c267b1e2e4644b60d809283b8b" gracePeriod=120
Sep 14 10:51:49.343457 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.343253    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="kube-rbac-proxy" containerID="cri-o://da681e6fdd5ce46924682c7c5c0b10239a86ffe52d08d89c406650bec91b2640" gracePeriod=120
Sep 14 10:51:49.343457 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.343307    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="prom-label-proxy" containerID="cri-o://688697e169da9b7c6dadac23c442c6efa162c44063e2a328c5ef2d3883149b83" gracePeriod=120
Sep 14 10:51:49.343457 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.343288    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="kube-rbac-proxy-web" containerID="cri-o://255595d8621498dec8db6f26af9b7945d5ab2542514bd1c36e014062dd220517" gracePeriod=120
Sep 14 10:51:49.343457 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.343381    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="config-reloader" containerID="cri-o://d6fa3257d6f0b2ea7ce2816a8cec67ceec6c7bfdfcb6121c9c844cb43e0078a1" gracePeriod=120
Sep 14 10:51:49.343457 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.343286    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="kube-rbac-proxy-metric" containerID="cri-o://8d620baf3f4d9331a08934f9626da568605678fdcf52fee9211532bfd77275df" gracePeriod=120
Sep 14 10:51:49.718918 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.718886    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerID="688697e169da9b7c6dadac23c442c6efa162c44063e2a328c5ef2d3883149b83" exitCode=0
Sep 14 10:51:49.718918 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.718911    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerID="8d620baf3f4d9331a08934f9626da568605678fdcf52fee9211532bfd77275df" exitCode=0
Sep 14 10:51:49.718918 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.718917    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerID="da681e6fdd5ce46924682c7c5c0b10239a86ffe52d08d89c406650bec91b2640" exitCode=0
Sep 14 10:51:49.718918 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.718923    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerID="d6fa3257d6f0b2ea7ce2816a8cec67ceec6c7bfdfcb6121c9c844cb43e0078a1" exitCode=0
Sep 14 10:51:49.719160 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.718930    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerID="7896b2353217ea5dfb49f3f1e08911c0d465f0c267b1e2e4644b60d809283b8b" exitCode=0
Sep 14 10:51:49.719160 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.718951    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerDied","Data":"688697e169da9b7c6dadac23c442c6efa162c44063e2a328c5ef2d3883149b83"}
Sep 14 10:51:49.719160 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.719019    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerDied","Data":"8d620baf3f4d9331a08934f9626da568605678fdcf52fee9211532bfd77275df"}
Sep 14 10:51:49.719160 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.719046    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerDied","Data":"da681e6fdd5ce46924682c7c5c0b10239a86ffe52d08d89c406650bec91b2640"}
Sep 14 10:51:49.719160 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.719071    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerDied","Data":"d6fa3257d6f0b2ea7ce2816a8cec67ceec6c7bfdfcb6121c9c844cb43e0078a1"}
Sep 14 10:51:49.719160 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:49.719095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerDied","Data":"7896b2353217ea5dfb49f3f1e08911c0d465f0c267b1e2e4644b60d809283b8b"}
Sep 14 10:51:50.595927 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.595903    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.702494 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702415    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy\") pod \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") "
Sep 14 10:51:50.702494 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702449    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") "
Sep 14 10:51:50.702494 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702470    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-cluster-tls-config\") pod \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") "
Sep 14 10:51:50.702756 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702510    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-alertmanager-trusted-ca-bundle\") pod \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") "
Sep 14 10:51:50.702756 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702534    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-main-tls\") pod \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") "
Sep 14 10:51:50.702756 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702561    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-volume\") pod \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") "
Sep 14 10:51:50.702756 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702609    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-alertmanager-main-db\") pod \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") "
Sep 14 10:51:50.702756 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702638    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-tls-assets\") pod \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") "
Sep 14 10:51:50.702756 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702670    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-out\") pod \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") "
Sep 14 10:51:50.702756 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702694    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dql44\" (UniqueName: \"kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-kube-api-access-dql44\") pod \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") "
Sep 14 10:51:50.702756 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702742    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-metrics-client-ca\") pod \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") "
Sep 14 10:51:50.703157 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702771    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-web-config\") pod \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") "
Sep 14 10:51:50.703157 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702796    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-web\") pod \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\" (UID: \"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d\") "
Sep 14 10:51:50.703157 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.702956    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:51:50.703157 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.703041    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-alertmanager-main-db\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:50.703406 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.703369    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:51:50.703493 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.703410    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:51:50.705124 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.705092    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:51:50.705487 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.705443    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:51:50.706118 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.706081    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:51:50.706250 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.706223    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-kube-api-access-dql44" (OuterVolumeSpecName: "kube-api-access-dql44") pod "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d"). InnerVolumeSpecName "kube-api-access-dql44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:51:50.707310 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.707282    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:51:50.707529 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.707509    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-volume" (OuterVolumeSpecName: "config-volume") pod "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:51:50.707798 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.707775    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:51:50.708336 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.708306    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-out" (OuterVolumeSpecName: "config-out") pod "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:51:50.709541 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.709520    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:51:50.715590 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.715569    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-web-config" (OuterVolumeSpecName: "web-config") pod "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" (UID: "7a2f546a-c63a-4c1b-af96-84dd7b25bc3d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:51:50.726486 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.726466    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerID="255595d8621498dec8db6f26af9b7945d5ab2542514bd1c36e014062dd220517" exitCode=0
Sep 14 10:51:50.726572 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.726511    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerDied","Data":"255595d8621498dec8db6f26af9b7945d5ab2542514bd1c36e014062dd220517"}
Sep 14 10:51:50.726572 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.726540    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.726572 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.726565    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a2f546a-c63a-4c1b-af96-84dd7b25bc3d","Type":"ContainerDied","Data":"1f77af44a9bf2ea788b8050dd5c39f8d05787a629588f94a492f8ebe0f7e46f8"}
Sep 14 10:51:50.726685 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.726597    2561 scope.go:117] "RemoveContainer" containerID="688697e169da9b7c6dadac23c442c6efa162c44063e2a328c5ef2d3883149b83"
Sep 14 10:51:50.735010 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.734995    2561 scope.go:117] "RemoveContainer" containerID="8d620baf3f4d9331a08934f9626da568605678fdcf52fee9211532bfd77275df"
Sep 14 10:51:50.741452 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.741438    2561 scope.go:117] "RemoveContainer" containerID="da681e6fdd5ce46924682c7c5c0b10239a86ffe52d08d89c406650bec91b2640"
Sep 14 10:51:50.747574 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.747559    2561 scope.go:117] "RemoveContainer" containerID="255595d8621498dec8db6f26af9b7945d5ab2542514bd1c36e014062dd220517"
Sep 14 10:51:50.753837 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.753818    2561 scope.go:117] "RemoveContainer" containerID="d6fa3257d6f0b2ea7ce2816a8cec67ceec6c7bfdfcb6121c9c844cb43e0078a1"
Sep 14 10:51:50.758558 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.758537    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:51:50.761111 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.761093    2561 scope.go:117] "RemoveContainer" containerID="7896b2353217ea5dfb49f3f1e08911c0d465f0c267b1e2e4644b60d809283b8b"
Sep 14 10:51:50.767807 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.767780    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:51:50.771743 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.771707    2561 scope.go:117] "RemoveContainer" containerID="b5a9c76037632622ef089ec8af41d55fff09ab73822c86c19ff10c99ad2bed8a"
Sep 14 10:51:50.778261 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.778240    2561 scope.go:117] "RemoveContainer" containerID="688697e169da9b7c6dadac23c442c6efa162c44063e2a328c5ef2d3883149b83"
Sep 14 10:51:50.778513 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:51:50.778495    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688697e169da9b7c6dadac23c442c6efa162c44063e2a328c5ef2d3883149b83\": container with ID starting with 688697e169da9b7c6dadac23c442c6efa162c44063e2a328c5ef2d3883149b83 not found: ID does not exist" containerID="688697e169da9b7c6dadac23c442c6efa162c44063e2a328c5ef2d3883149b83"
Sep 14 10:51:50.778564 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.778519    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688697e169da9b7c6dadac23c442c6efa162c44063e2a328c5ef2d3883149b83"} err="failed to get container status \"688697e169da9b7c6dadac23c442c6efa162c44063e2a328c5ef2d3883149b83\": rpc error: code = NotFound desc = could not find container \"688697e169da9b7c6dadac23c442c6efa162c44063e2a328c5ef2d3883149b83\": container with ID starting with 688697e169da9b7c6dadac23c442c6efa162c44063e2a328c5ef2d3883149b83 not found: ID does not exist"
Sep 14 10:51:50.778564 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.778535    2561 scope.go:117] "RemoveContainer" containerID="8d620baf3f4d9331a08934f9626da568605678fdcf52fee9211532bfd77275df"
Sep 14 10:51:50.778812 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:51:50.778793    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d620baf3f4d9331a08934f9626da568605678fdcf52fee9211532bfd77275df\": container with ID starting with 8d620baf3f4d9331a08934f9626da568605678fdcf52fee9211532bfd77275df not found: ID does not exist" containerID="8d620baf3f4d9331a08934f9626da568605678fdcf52fee9211532bfd77275df"
Sep 14 10:51:50.778877 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.778817    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d620baf3f4d9331a08934f9626da568605678fdcf52fee9211532bfd77275df"} err="failed to get container status \"8d620baf3f4d9331a08934f9626da568605678fdcf52fee9211532bfd77275df\": rpc error: code = NotFound desc = could not find container \"8d620baf3f4d9331a08934f9626da568605678fdcf52fee9211532bfd77275df\": container with ID starting with 8d620baf3f4d9331a08934f9626da568605678fdcf52fee9211532bfd77275df not found: ID does not exist"
Sep 14 10:51:50.778877 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.778832    2561 scope.go:117] "RemoveContainer" containerID="da681e6fdd5ce46924682c7c5c0b10239a86ffe52d08d89c406650bec91b2640"
Sep 14 10:51:50.779082 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:51:50.779067    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da681e6fdd5ce46924682c7c5c0b10239a86ffe52d08d89c406650bec91b2640\": container with ID starting with da681e6fdd5ce46924682c7c5c0b10239a86ffe52d08d89c406650bec91b2640 not found: ID does not exist" containerID="da681e6fdd5ce46924682c7c5c0b10239a86ffe52d08d89c406650bec91b2640"
Sep 14 10:51:50.779130 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.779084    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da681e6fdd5ce46924682c7c5c0b10239a86ffe52d08d89c406650bec91b2640"} err="failed to get container status \"da681e6fdd5ce46924682c7c5c0b10239a86ffe52d08d89c406650bec91b2640\": rpc error: code = NotFound desc = could not find container \"da681e6fdd5ce46924682c7c5c0b10239a86ffe52d08d89c406650bec91b2640\": container with ID starting with da681e6fdd5ce46924682c7c5c0b10239a86ffe52d08d89c406650bec91b2640 not found: ID does not exist"
Sep 14 10:51:50.779130 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.779096    2561 scope.go:117] "RemoveContainer" containerID="255595d8621498dec8db6f26af9b7945d5ab2542514bd1c36e014062dd220517"
Sep 14 10:51:50.779314 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:51:50.779299    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255595d8621498dec8db6f26af9b7945d5ab2542514bd1c36e014062dd220517\": container with ID starting with 255595d8621498dec8db6f26af9b7945d5ab2542514bd1c36e014062dd220517 not found: ID does not exist" containerID="255595d8621498dec8db6f26af9b7945d5ab2542514bd1c36e014062dd220517"
Sep 14 10:51:50.779353 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.779316    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255595d8621498dec8db6f26af9b7945d5ab2542514bd1c36e014062dd220517"} err="failed to get container status \"255595d8621498dec8db6f26af9b7945d5ab2542514bd1c36e014062dd220517\": rpc error: code = NotFound desc = could not find container \"255595d8621498dec8db6f26af9b7945d5ab2542514bd1c36e014062dd220517\": container with ID starting with 255595d8621498dec8db6f26af9b7945d5ab2542514bd1c36e014062dd220517 not found: ID does not exist"
Sep 14 10:51:50.779353 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.779327    2561 scope.go:117] "RemoveContainer" containerID="d6fa3257d6f0b2ea7ce2816a8cec67ceec6c7bfdfcb6121c9c844cb43e0078a1"
Sep 14 10:51:50.779539 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:51:50.779526    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6fa3257d6f0b2ea7ce2816a8cec67ceec6c7bfdfcb6121c9c844cb43e0078a1\": container with ID starting with d6fa3257d6f0b2ea7ce2816a8cec67ceec6c7bfdfcb6121c9c844cb43e0078a1 not found: ID does not exist" containerID="d6fa3257d6f0b2ea7ce2816a8cec67ceec6c7bfdfcb6121c9c844cb43e0078a1"
Sep 14 10:51:50.779573 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.779541    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fa3257d6f0b2ea7ce2816a8cec67ceec6c7bfdfcb6121c9c844cb43e0078a1"} err="failed to get container status \"d6fa3257d6f0b2ea7ce2816a8cec67ceec6c7bfdfcb6121c9c844cb43e0078a1\": rpc error: code = NotFound desc = could not find container \"d6fa3257d6f0b2ea7ce2816a8cec67ceec6c7bfdfcb6121c9c844cb43e0078a1\": container with ID starting with d6fa3257d6f0b2ea7ce2816a8cec67ceec6c7bfdfcb6121c9c844cb43e0078a1 not found: ID does not exist"
Sep 14 10:51:50.779573 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.779552    2561 scope.go:117] "RemoveContainer" containerID="7896b2353217ea5dfb49f3f1e08911c0d465f0c267b1e2e4644b60d809283b8b"
Sep 14 10:51:50.779750 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:51:50.779734    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7896b2353217ea5dfb49f3f1e08911c0d465f0c267b1e2e4644b60d809283b8b\": container with ID starting with 7896b2353217ea5dfb49f3f1e08911c0d465f0c267b1e2e4644b60d809283b8b not found: ID does not exist" containerID="7896b2353217ea5dfb49f3f1e08911c0d465f0c267b1e2e4644b60d809283b8b"
Sep 14 10:51:50.779785 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.779753    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7896b2353217ea5dfb49f3f1e08911c0d465f0c267b1e2e4644b60d809283b8b"} err="failed to get container status \"7896b2353217ea5dfb49f3f1e08911c0d465f0c267b1e2e4644b60d809283b8b\": rpc error: code = NotFound desc = could not find container \"7896b2353217ea5dfb49f3f1e08911c0d465f0c267b1e2e4644b60d809283b8b\": container with ID starting with 7896b2353217ea5dfb49f3f1e08911c0d465f0c267b1e2e4644b60d809283b8b not found: ID does not exist"
Sep 14 10:51:50.779785 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.779765    2561 scope.go:117] "RemoveContainer" containerID="b5a9c76037632622ef089ec8af41d55fff09ab73822c86c19ff10c99ad2bed8a"
Sep 14 10:51:50.779968 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:51:50.779954    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a9c76037632622ef089ec8af41d55fff09ab73822c86c19ff10c99ad2bed8a\": container with ID starting with b5a9c76037632622ef089ec8af41d55fff09ab73822c86c19ff10c99ad2bed8a not found: ID does not exist" containerID="b5a9c76037632622ef089ec8af41d55fff09ab73822c86c19ff10c99ad2bed8a"
Sep 14 10:51:50.780007 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.779970    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a9c76037632622ef089ec8af41d55fff09ab73822c86c19ff10c99ad2bed8a"} err="failed to get container status \"b5a9c76037632622ef089ec8af41d55fff09ab73822c86c19ff10c99ad2bed8a\": rpc error: code = NotFound desc = could not find container \"b5a9c76037632622ef089ec8af41d55fff09ab73822c86c19ff10c99ad2bed8a\": container with ID starting with b5a9c76037632622ef089ec8af41d55fff09ab73822c86c19ff10c99ad2bed8a not found: ID does not exist"
Sep 14 10:51:50.803930 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.803911    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:50.803930 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.803929    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:50.804050 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.803940    2561 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-cluster-tls-config\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:50.804050 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.803949    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:50.804050 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.803958    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-main-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:50.804050 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.803966    2561 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:50.804050 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.803974    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-tls-assets\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:50.804050 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.803981    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-config-out\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:50.804050 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.803989    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dql44\" (UniqueName: \"kubernetes.io/projected/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-kube-api-access-dql44\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:50.804050 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.803997    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-metrics-client-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:50.804050 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.804031    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-web-config\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:50.804050 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.804040    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:51:50.813088 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813069    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:51:50.813330 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813318    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b44ae89-526e-4b83-a81b-85e9e9deb339" containerName="console"
Sep 14 10:51:50.813370 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813332    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b44ae89-526e-4b83-a81b-85e9e9deb339" containerName="console"
Sep 14 10:51:50.813370 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813343    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="kube-rbac-proxy-metric"
Sep 14 10:51:50.813370 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813348    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="kube-rbac-proxy-metric"
Sep 14 10:51:50.813370 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813357    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="kube-rbac-proxy-web"
Sep 14 10:51:50.813370 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813362    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="kube-rbac-proxy-web"
Sep 14 10:51:50.813370 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813368    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="init-config-reloader"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813373    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="init-config-reloader"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813382    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="prom-label-proxy"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813388    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="prom-label-proxy"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813394    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="kube-rbac-proxy"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813399    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="kube-rbac-proxy"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813406    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="alertmanager"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813411    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="alertmanager"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813417    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="config-reloader"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813422    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="config-reloader"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813464    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b44ae89-526e-4b83-a81b-85e9e9deb339" containerName="console"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813471    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="kube-rbac-proxy-metric"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813477    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="prom-label-proxy"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813483    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="config-reloader"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813488    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="kube-rbac-proxy-web"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813494    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="kube-rbac-proxy"
Sep 14 10:51:50.813546 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.813499    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" containerName="alertmanager"
Sep 14 10:51:50.818270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.818253    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.822801 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.822782    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 14 10:51:50.823029 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.823014    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 14 10:51:50.823129 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.823114    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 14 10:51:50.823291 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.823274    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 14 10:51:50.823378 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.823362    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 14 10:51:50.823752 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.823738    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 14 10:51:50.823892 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.823881    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 14 10:51:50.825174 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.825152    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 14 10:51:50.825260 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.825235    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-84qd7\""
Sep 14 10:51:50.828817 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.828801    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 14 10:51:50.856198 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.856179    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:51:50.905129 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.905103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd05f595-b3de-48b3-b932-c29f72a7a287-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.905229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.905135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.905229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.905164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.905229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.905216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.905359 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.905278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd05f595-b3de-48b3-b932-c29f72a7a287-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.905359 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.905343    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd05f595-b3de-48b3-b932-c29f72a7a287-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.905435 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.905366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-config-volume\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.905435 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.905390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd05f595-b3de-48b3-b932-c29f72a7a287-config-out\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.905435 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.905410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd05f595-b3de-48b3-b932-c29f72a7a287-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.905544 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.905466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.905544 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.905498    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xjhd\" (UniqueName: \"kubernetes.io/projected/cd05f595-b3de-48b3-b932-c29f72a7a287-kube-api-access-7xjhd\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.905544 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.905521    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:50.905544 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:50.905539    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-web-config\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.006077 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.006001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xjhd\" (UniqueName: \"kubernetes.io/projected/cd05f595-b3de-48b3-b932-c29f72a7a287-kube-api-access-7xjhd\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.006077 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.006032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.006077 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.006052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-web-config\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.006077 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.006068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd05f595-b3de-48b3-b932-c29f72a7a287-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.006374 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.006209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.006374 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.006257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.006374 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.006283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.006521 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.006408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd05f595-b3de-48b3-b932-c29f72a7a287-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.006521 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.006471    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd05f595-b3de-48b3-b932-c29f72a7a287-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.006521 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.006505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-config-volume\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.006668 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.006544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd05f595-b3de-48b3-b932-c29f72a7a287-config-out\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.006668 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.006581    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd05f595-b3de-48b3-b932-c29f72a7a287-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.006668 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.006619    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.009134 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.009066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd05f595-b3de-48b3-b932-c29f72a7a287-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.009246 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.009229    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.009321 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.009241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-web-config\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.009321 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.009245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.009429 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.009413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.009429 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.009420    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-config-volume\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.009565 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.009538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd05f595-b3de-48b3-b932-c29f72a7a287-config-out\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.009565 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.009551    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd05f595-b3de-48b3-b932-c29f72a7a287-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.009685 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.009654    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd05f595-b3de-48b3-b932-c29f72a7a287-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.009752 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.009738    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.010149 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.010128    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd05f595-b3de-48b3-b932-c29f72a7a287-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.011164 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.011142    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd05f595-b3de-48b3-b932-c29f72a7a287-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.016899 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.016877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xjhd\" (UniqueName: \"kubernetes.io/projected/cd05f595-b3de-48b3-b932-c29f72a7a287-kube-api-access-7xjhd\") pod \"alertmanager-main-0\" (UID: \"cd05f595-b3de-48b3-b932-c29f72a7a287\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.128363 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.128340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:51:51.412949 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.412924    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:51:51.414700 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:51:51.414663    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd05f595_b3de_48b3_b932_c29f72a7a287.slice/crio-ff9aef57ebec5c85395e005d4a859b7aef6bf1a236b5a8f302780289cdb9fdec WatchSource:0}: Error finding container ff9aef57ebec5c85395e005d4a859b7aef6bf1a236b5a8f302780289cdb9fdec: Status 404 returned error can't find the container with id ff9aef57ebec5c85395e005d4a859b7aef6bf1a236b5a8f302780289cdb9fdec
Sep 14 10:51:51.732225 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.732140    2561 generic.go:356] "Generic (PLEG): container finished" podID="cd05f595-b3de-48b3-b932-c29f72a7a287" containerID="4ae92a87f91503825fea8ef2103a728386247ad403f771f8e48caec8beea2775" exitCode=0
Sep 14 10:51:51.732225 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.732217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd05f595-b3de-48b3-b932-c29f72a7a287","Type":"ContainerDied","Data":"4ae92a87f91503825fea8ef2103a728386247ad403f771f8e48caec8beea2775"}
Sep 14 10:51:51.732589 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.732283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd05f595-b3de-48b3-b932-c29f72a7a287","Type":"ContainerStarted","Data":"ff9aef57ebec5c85395e005d4a859b7aef6bf1a236b5a8f302780289cdb9fdec"}
Sep 14 10:51:51.801823 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:51.801801    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a2f546a-c63a-4c1b-af96-84dd7b25bc3d" path="/var/lib/kubelet/pods/7a2f546a-c63a-4c1b-af96-84dd7b25bc3d/volumes"
Sep 14 10:51:52.739087 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:52.739054    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd05f595-b3de-48b3-b932-c29f72a7a287","Type":"ContainerStarted","Data":"4b359d06656544a8ee42ab1984f90d11efc83c3e36d8b0f11d44fd26e30e4662"}
Sep 14 10:51:52.739488 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:52.739114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd05f595-b3de-48b3-b932-c29f72a7a287","Type":"ContainerStarted","Data":"606ff97bb7853ace84eabee4d7e22c950d56edf0afddfdc968a0bab505a91024"}
Sep 14 10:51:52.739488 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:52.739142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd05f595-b3de-48b3-b932-c29f72a7a287","Type":"ContainerStarted","Data":"23e66b8a4b1bdffeb92044da17bbe1a0bc29654bbdfa15f12aa17f4cedd1330c"}
Sep 14 10:51:52.739488 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:52.739167    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd05f595-b3de-48b3-b932-c29f72a7a287","Type":"ContainerStarted","Data":"5d465851759bdc4634c2475836c348789fa47b9f81de423291e6a5623f406854"}
Sep 14 10:51:52.739488 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:52.739192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd05f595-b3de-48b3-b932-c29f72a7a287","Type":"ContainerStarted","Data":"ce7f4815b1bcaf790a1e43f64d79ab05777bfe53b534b343c397f1056109e7ac"}
Sep 14 10:51:52.739488 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:52.739214    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd05f595-b3de-48b3-b932-c29f72a7a287","Type":"ContainerStarted","Data":"7b14dcb20c98a623cdebe93c2172b3c75e299dc557b1f71ff3ee20b75c1062f2"}
Sep 14 10:51:52.771325 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:52.771273    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.755607145 podStartE2EDuration="2.771256999s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.649854ms" isStatefulPod=false podCreationTimestamp="2026-09-14 10:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:51:52.769392019 +0000 UTC m=+217.454076445" watchObservedRunningTime="2026-09-14 10:51:52.771256999 +0000 UTC m=+217.455941387"
Sep 14 10:51:53.331271 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.331233    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"]
Sep 14 10:51:53.334595 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.334569    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.338004 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.337978    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 14 10:51:53.338309 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.338291    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-58rb5\""
Sep 14 10:51:53.338642 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.338627    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 14 10:51:53.338774 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.338731    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 14 10:51:53.339173 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.339158    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 14 10:51:53.339245 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.339165    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 14 10:51:53.345953 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.345935    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 14 10:51:53.365967 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.365942    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"]
Sep 14 10:51:53.428355 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.428328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.428459 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.428369    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-telemeter-client-tls\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.428459 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.428397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-serving-certs-ca-bundle\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.428544 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.428474    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.428544 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.428519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-metrics-client-ca\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.428544 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.428541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-federate-client-tls\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.428656 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.428595    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-secret-telemeter-client\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.428656 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.428621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88nw\" (UniqueName: \"kubernetes.io/projected/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-kube-api-access-b88nw\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.529554 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.529526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-federate-client-tls\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.529675 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.529579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-secret-telemeter-client\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.529675 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.529614    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88nw\" (UniqueName: \"kubernetes.io/projected/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-kube-api-access-b88nw\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.529675 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.529640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.529675 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.529671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-telemeter-client-tls\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.529913 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.529699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-serving-certs-ca-bundle\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.529913 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.529764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.529913 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.529901    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-metrics-client-ca\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.530525 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.530497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-serving-certs-ca-bundle\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.530644 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.530563    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-metrics-client-ca\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.530893 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.530869    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.532342 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.532319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-federate-client-tls\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.532456 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.532335    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.532573 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.532557    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-secret-telemeter-client\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.532607 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.532588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-telemeter-client-tls\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.538752 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.538733    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88nw\" (UniqueName: \"kubernetes.io/projected/4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6-kube-api-access-b88nw\") pod \"telemeter-client-688f8d78d8-mrd7g\" (UID: \"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6\") " pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.644432 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.644410    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"
Sep 14 10:51:53.919433 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:53.919408    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-688f8d78d8-mrd7g"]
Sep 14 10:51:53.921184 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:51:53.921158    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f57ac5a_74bd_4ea4_9851_1ae3e9c4dcf6.slice/crio-f9248b7564876d7bb7db1396505dd161350f76b7e26632fcf4d5eeebaa72777a WatchSource:0}: Error finding container f9248b7564876d7bb7db1396505dd161350f76b7e26632fcf4d5eeebaa72777a: Status 404 returned error can't find the container with id f9248b7564876d7bb7db1396505dd161350f76b7e26632fcf4d5eeebaa72777a
Sep 14 10:51:54.751260 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:54.751215    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g" event={"ID":"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6","Type":"ContainerStarted","Data":"f9248b7564876d7bb7db1396505dd161350f76b7e26632fcf4d5eeebaa72777a"}
Sep 14 10:51:56.762899 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.762863    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g" event={"ID":"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6","Type":"ContainerStarted","Data":"f2632ecccc55fc0619d6acb3dce8758c19d82fa7e35691936349d6f430214231"}
Sep 14 10:51:56.763262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.762964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g" event={"ID":"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6","Type":"ContainerStarted","Data":"69475524fdf13086a759e991a2661d8ec9719f87d7c8fad6dfef8b6f35757c50"}
Sep 14 10:51:56.763262 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.762985    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g" event={"ID":"4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6","Type":"ContainerStarted","Data":"8123cb9e272b138ccf7e0ca3cf0201e682b84e155b2ef4a796b2ed0d3ae9b451"}
Sep 14 10:51:56.788472 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.788429    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-688f8d78d8-mrd7g" podStartSLOduration=1.846631425 podStartE2EDuration="3.78841506s" totalImagesPullingTime="1.941783635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:51:56.784613986 +0000 UTC m=+221.469298376" watchObservedRunningTime="2026-09-14 10:51:56.78841506 +0000 UTC m=+221.473099449"
Sep 14 10:51:56.871652 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.871625    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bdff6c9f4-8nnqz"]
Sep 14 10:51:56.874967 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.874946    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:56.877363 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.877346    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-75bjs\""
Sep 14 10:51:56.878533 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.878502    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 14 10:51:56.878617 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.878577    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 14 10:51:56.878924 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.878891    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 14 10:51:56.879213 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.879200    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 14 10:51:56.879250 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.879228    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 14 10:51:56.885143 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.885127    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 14 10:51:56.887919 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.887894    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bdff6c9f4-8nnqz"]
Sep 14 10:51:56.957352 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.957310    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-oauth-serving-cert\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:56.957352 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.957361    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e91f309-b591-4924-aee0-0c34c55a45ec-console-serving-cert\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:56.957504 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.957378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjg5\" (UniqueName: \"kubernetes.io/projected/7e91f309-b591-4924-aee0-0c34c55a45ec-kube-api-access-jqjg5\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:56.957504 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.957403    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e91f309-b591-4924-aee0-0c34c55a45ec-console-oauth-config\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:56.957504 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.957438    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-trusted-ca-bundle\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:56.957601 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.957506    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-console-config\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:56.957601 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:56.957524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-service-ca\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.058650 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.058581    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-console-config\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.058650 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.058621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-service-ca\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.058837 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.058659    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-oauth-serving-cert\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.058837 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.058691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e91f309-b591-4924-aee0-0c34c55a45ec-console-serving-cert\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.058837 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.058725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjg5\" (UniqueName: \"kubernetes.io/projected/7e91f309-b591-4924-aee0-0c34c55a45ec-kube-api-access-jqjg5\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.059094 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.059071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e91f309-b591-4924-aee0-0c34c55a45ec-console-oauth-config\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.059230 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.059135    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-trusted-ca-bundle\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.059385 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.059368    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-oauth-serving-cert\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.059442 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.059384    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-console-config\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.059524 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.059498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-service-ca\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.059937 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.059918    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-trusted-ca-bundle\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.061125 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.061107    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e91f309-b591-4924-aee0-0c34c55a45ec-console-serving-cert\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.061195 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.061187    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e91f309-b591-4924-aee0-0c34c55a45ec-console-oauth-config\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.067528 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.067512    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjg5\" (UniqueName: \"kubernetes.io/projected/7e91f309-b591-4924-aee0-0c34c55a45ec-kube-api-access-jqjg5\") pod \"console-7bdff6c9f4-8nnqz\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") " pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.185327 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.185285    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:51:57.459079 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.459052    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bdff6c9f4-8nnqz"]
Sep 14 10:51:57.461658 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:51:57.461626    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e91f309_b591_4924_aee0_0c34c55a45ec.slice/crio-83b9b281428ea3731489bc8930810c3c8ae229c27557829097c77305c6f09a2c WatchSource:0}: Error finding container 83b9b281428ea3731489bc8930810c3c8ae229c27557829097c77305c6f09a2c: Status 404 returned error can't find the container with id 83b9b281428ea3731489bc8930810c3c8ae229c27557829097c77305c6f09a2c
Sep 14 10:51:57.768815 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.768706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdff6c9f4-8nnqz" event={"ID":"7e91f309-b591-4924-aee0-0c34c55a45ec","Type":"ContainerStarted","Data":"85d36bcd1b0e693f89a98734e3e492fbedeae04b70ec4894e3ee966d5a8a1a43"}
Sep 14 10:51:57.768815 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.768783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdff6c9f4-8nnqz" event={"ID":"7e91f309-b591-4924-aee0-0c34c55a45ec","Type":"ContainerStarted","Data":"83b9b281428ea3731489bc8930810c3c8ae229c27557829097c77305c6f09a2c"}
Sep 14 10:51:57.788037 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:51:57.787986    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bdff6c9f4-8nnqz" podStartSLOduration=1.7879705609999998 podStartE2EDuration="1.787970561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:51:57.787220736 +0000 UTC m=+222.471905124" watchObservedRunningTime="2026-09-14 10:51:57.787970561 +0000 UTC m=+222.472654950"
Sep 14 10:52:07.186006 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:52:07.185959    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:52:07.186480 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:52:07.186034    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:52:07.190660 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:52:07.190639    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:52:07.808212 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:52:07.808178    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:53:07.668140 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.668102    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-748f74fdbf-59rdf"]
Sep 14 10:53:07.671328 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.671307    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.689880 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.689855    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-748f74fdbf-59rdf"]
Sep 14 10:53:07.784410 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.784378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-config\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.784410 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.784415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-oauth-serving-cert\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.784627 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.784432    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-trusted-ca-bundle\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.784627 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.784507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8kww\" (UniqueName: \"kubernetes.io/projected/acedcbd0-81ff-494e-ab84-de490f2e31d0-kube-api-access-r8kww\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.784627 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.784543    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-serving-cert\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.784627 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.784566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-oauth-config\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.784821 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.784650    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-service-ca\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.885177 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.885138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-serving-cert\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.885177 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.885183    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-oauth-config\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.885432 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.885211    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-service-ca\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.885432 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.885250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-config\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.885432 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.885273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-oauth-serving-cert\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.885432 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.885288    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-trusted-ca-bundle\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.885432 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.885333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8kww\" (UniqueName: \"kubernetes.io/projected/acedcbd0-81ff-494e-ab84-de490f2e31d0-kube-api-access-r8kww\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.886147 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.886118    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-config\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.886269 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.886118    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-service-ca\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.886269 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.886225    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-trusted-ca-bundle\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.886269 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.886242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-oauth-serving-cert\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.887667 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.887648    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-serving-cert\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.887779 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.887667    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-oauth-config\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.895350 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.895330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8kww\" (UniqueName: \"kubernetes.io/projected/acedcbd0-81ff-494e-ab84-de490f2e31d0-kube-api-access-r8kww\") pod \"console-748f74fdbf-59rdf\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") " pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:07.980183 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:07.980125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:08.252253 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:08.252024    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-748f74fdbf-59rdf"]
Sep 14 10:53:08.254887 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:53:08.254860    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacedcbd0_81ff_494e_ab84_de490f2e31d0.slice/crio-6d6e767db5ff371502a436eb11ac40c73f4bab139d467a686289428c7937b5bd WatchSource:0}: Error finding container 6d6e767db5ff371502a436eb11ac40c73f4bab139d467a686289428c7937b5bd: Status 404 returned error can't find the container with id 6d6e767db5ff371502a436eb11ac40c73f4bab139d467a686289428c7937b5bd
Sep 14 10:53:09.027336 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:09.027302    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748f74fdbf-59rdf" event={"ID":"acedcbd0-81ff-494e-ab84-de490f2e31d0","Type":"ContainerStarted","Data":"a2ce1b287a06c52724943caee326e79645777ad5f0b9b7aa2ca8dbbca0efbfee"}
Sep 14 10:53:09.027692 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:09.027349    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748f74fdbf-59rdf" event={"ID":"acedcbd0-81ff-494e-ab84-de490f2e31d0","Type":"ContainerStarted","Data":"6d6e767db5ff371502a436eb11ac40c73f4bab139d467a686289428c7937b5bd"}
Sep 14 10:53:09.048218 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:09.048173    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-748f74fdbf-59rdf" podStartSLOduration=2.048159061 podStartE2EDuration="2.048159061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:53:09.046952032 +0000 UTC m=+293.731636433" watchObservedRunningTime="2026-09-14 10:53:09.048159061 +0000 UTC m=+293.732843451"
Sep 14 10:53:15.742017 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:15.741985    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 10:53:15.748099 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:15.748078    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 10:53:15.752340 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:15.752320    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 14 10:53:17.981073 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:17.981038    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:17.983403 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:17.981096    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:17.985700 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:17.985673    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:18.064630 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:18.064603    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:53:18.124060 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:18.124023    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bdff6c9f4-8nnqz"]
Sep 14 10:53:43.144761 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.144688    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bdff6c9f4-8nnqz" podUID="7e91f309-b591-4924-aee0-0c34c55a45ec" containerName="console" containerID="cri-o://85d36bcd1b0e693f89a98734e3e492fbedeae04b70ec4894e3ee966d5a8a1a43" gracePeriod=15
Sep 14 10:53:43.377510 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.377488    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:53:43.443577 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.443516    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqjg5\" (UniqueName: \"kubernetes.io/projected/7e91f309-b591-4924-aee0-0c34c55a45ec-kube-api-access-jqjg5\") pod \"7e91f309-b591-4924-aee0-0c34c55a45ec\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") "
Sep 14 10:53:43.443577 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.443545    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-service-ca\") pod \"7e91f309-b591-4924-aee0-0c34c55a45ec\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") "
Sep 14 10:53:43.443577 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.443570    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e91f309-b591-4924-aee0-0c34c55a45ec-console-serving-cert\") pod \"7e91f309-b591-4924-aee0-0c34c55a45ec\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") "
Sep 14 10:53:43.443844 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.443588    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e91f309-b591-4924-aee0-0c34c55a45ec-console-oauth-config\") pod \"7e91f309-b591-4924-aee0-0c34c55a45ec\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") "
Sep 14 10:53:43.443844 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.443686    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-oauth-serving-cert\") pod \"7e91f309-b591-4924-aee0-0c34c55a45ec\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") "
Sep 14 10:53:43.443844 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.443746    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-trusted-ca-bundle\") pod \"7e91f309-b591-4924-aee0-0c34c55a45ec\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") "
Sep 14 10:53:43.443844 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.443786    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-console-config\") pod \"7e91f309-b591-4924-aee0-0c34c55a45ec\" (UID: \"7e91f309-b591-4924-aee0-0c34c55a45ec\") "
Sep 14 10:53:43.444078 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.443929    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-service-ca" (OuterVolumeSpecName: "service-ca") pod "7e91f309-b591-4924-aee0-0c34c55a45ec" (UID: "7e91f309-b591-4924-aee0-0c34c55a45ec"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:53:43.444138 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.444075    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:53:43.444138 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.444107    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7e91f309-b591-4924-aee0-0c34c55a45ec" (UID: "7e91f309-b591-4924-aee0-0c34c55a45ec"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:53:43.444239 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.444190    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7e91f309-b591-4924-aee0-0c34c55a45ec" (UID: "7e91f309-b591-4924-aee0-0c34c55a45ec"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:53:43.444239 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.444222    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-console-config" (OuterVolumeSpecName: "console-config") pod "7e91f309-b591-4924-aee0-0c34c55a45ec" (UID: "7e91f309-b591-4924-aee0-0c34c55a45ec"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:53:43.445666 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.445638    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e91f309-b591-4924-aee0-0c34c55a45ec-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7e91f309-b591-4924-aee0-0c34c55a45ec" (UID: "7e91f309-b591-4924-aee0-0c34c55a45ec"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:53:43.445666 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.445640    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e91f309-b591-4924-aee0-0c34c55a45ec-kube-api-access-jqjg5" (OuterVolumeSpecName: "kube-api-access-jqjg5") pod "7e91f309-b591-4924-aee0-0c34c55a45ec" (UID: "7e91f309-b591-4924-aee0-0c34c55a45ec"). InnerVolumeSpecName "kube-api-access-jqjg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:53:43.445851 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.445663    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e91f309-b591-4924-aee0-0c34c55a45ec-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7e91f309-b591-4924-aee0-0c34c55a45ec" (UID: "7e91f309-b591-4924-aee0-0c34c55a45ec"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:53:43.544399 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.544380    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-oauth-serving-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:53:43.544399 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.544398    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-trusted-ca-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:53:43.544540 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.544408    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e91f309-b591-4924-aee0-0c34c55a45ec-console-config\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:53:43.544540 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.544416    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqjg5\" (UniqueName: \"kubernetes.io/projected/7e91f309-b591-4924-aee0-0c34c55a45ec-kube-api-access-jqjg5\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:53:43.544540 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.544425    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e91f309-b591-4924-aee0-0c34c55a45ec-console-serving-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:53:43.544540 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:43.544433    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e91f309-b591-4924-aee0-0c34c55a45ec-console-oauth-config\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:53:44.157771 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:44.157742    2561 generic.go:356] "Generic (PLEG): container finished" podID="7e91f309-b591-4924-aee0-0c34c55a45ec" containerID="85d36bcd1b0e693f89a98734e3e492fbedeae04b70ec4894e3ee966d5a8a1a43" exitCode=0
Sep 14 10:53:44.158137 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:44.157786    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdff6c9f4-8nnqz" event={"ID":"7e91f309-b591-4924-aee0-0c34c55a45ec","Type":"ContainerDied","Data":"85d36bcd1b0e693f89a98734e3e492fbedeae04b70ec4894e3ee966d5a8a1a43"}
Sep 14 10:53:44.158137 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:44.157830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdff6c9f4-8nnqz" event={"ID":"7e91f309-b591-4924-aee0-0c34c55a45ec","Type":"ContainerDied","Data":"83b9b281428ea3731489bc8930810c3c8ae229c27557829097c77305c6f09a2c"}
Sep 14 10:53:44.158137 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:44.157833    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdff6c9f4-8nnqz"
Sep 14 10:53:44.158137 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:44.157853    2561 scope.go:117] "RemoveContainer" containerID="85d36bcd1b0e693f89a98734e3e492fbedeae04b70ec4894e3ee966d5a8a1a43"
Sep 14 10:53:44.165741 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:44.165726    2561 scope.go:117] "RemoveContainer" containerID="85d36bcd1b0e693f89a98734e3e492fbedeae04b70ec4894e3ee966d5a8a1a43"
Sep 14 10:53:44.166008 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:53:44.165987    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d36bcd1b0e693f89a98734e3e492fbedeae04b70ec4894e3ee966d5a8a1a43\": container with ID starting with 85d36bcd1b0e693f89a98734e3e492fbedeae04b70ec4894e3ee966d5a8a1a43 not found: ID does not exist" containerID="85d36bcd1b0e693f89a98734e3e492fbedeae04b70ec4894e3ee966d5a8a1a43"
Sep 14 10:53:44.166054 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:44.166016    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d36bcd1b0e693f89a98734e3e492fbedeae04b70ec4894e3ee966d5a8a1a43"} err="failed to get container status \"85d36bcd1b0e693f89a98734e3e492fbedeae04b70ec4894e3ee966d5a8a1a43\": rpc error: code = NotFound desc = could not find container \"85d36bcd1b0e693f89a98734e3e492fbedeae04b70ec4894e3ee966d5a8a1a43\": container with ID starting with 85d36bcd1b0e693f89a98734e3e492fbedeae04b70ec4894e3ee966d5a8a1a43 not found: ID does not exist"
Sep 14 10:53:44.190968 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:44.190944    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bdff6c9f4-8nnqz"]
Sep 14 10:53:44.195067 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:44.195047    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bdff6c9f4-8nnqz"]
Sep 14 10:53:45.801690 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:53:45.801661    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e91f309-b591-4924-aee0-0c34c55a45ec" path="/var/lib/kubelet/pods/7e91f309-b591-4924-aee0-0c34c55a45ec/volumes"
Sep 14 10:54:18.108375 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.108343    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"]
Sep 14 10:54:18.108779 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.108652    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e91f309-b591-4924-aee0-0c34c55a45ec" containerName="console"
Sep 14 10:54:18.108779 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.108664    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e91f309-b591-4924-aee0-0c34c55a45ec" containerName="console"
Sep 14 10:54:18.108779 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.108756    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e91f309-b591-4924-aee0-0c34c55a45ec" containerName="console"
Sep 14 10:54:18.112866 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.112849    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"
Sep 14 10:54:18.115601 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.115582    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 10:54:18.116080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.116067    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 10:54:18.117809 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.117795    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8p4wd\""
Sep 14 10:54:18.147960 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.147936    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"]
Sep 14 10:54:18.207773 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.207706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ff18e7-0c25-47e3-8a97-a1f92618d723-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk\" (UID: \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"
Sep 14 10:54:18.207773 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.207768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2qx7\" (UniqueName: \"kubernetes.io/projected/e7ff18e7-0c25-47e3-8a97-a1f92618d723-kube-api-access-l2qx7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk\" (UID: \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"
Sep 14 10:54:18.207929 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.207787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ff18e7-0c25-47e3-8a97-a1f92618d723-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk\" (UID: \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"
Sep 14 10:54:18.308998 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.308975    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ff18e7-0c25-47e3-8a97-a1f92618d723-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk\" (UID: \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"
Sep 14 10:54:18.309114 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.309016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2qx7\" (UniqueName: \"kubernetes.io/projected/e7ff18e7-0c25-47e3-8a97-a1f92618d723-kube-api-access-l2qx7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk\" (UID: \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"
Sep 14 10:54:18.309114 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.309042    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ff18e7-0c25-47e3-8a97-a1f92618d723-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk\" (UID: \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"
Sep 14 10:54:18.309405 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.309383    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ff18e7-0c25-47e3-8a97-a1f92618d723-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk\" (UID: \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"
Sep 14 10:54:18.309444 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.309399    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ff18e7-0c25-47e3-8a97-a1f92618d723-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk\" (UID: \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"
Sep 14 10:54:18.320484 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.320455    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2qx7\" (UniqueName: \"kubernetes.io/projected/e7ff18e7-0c25-47e3-8a97-a1f92618d723-kube-api-access-l2qx7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk\" (UID: \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"
Sep 14 10:54:18.421413 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.421351    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"
Sep 14 10:54:18.701361 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.701339    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"]
Sep 14 10:54:18.702184 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:54:18.702161    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ff18e7_0c25_47e3_8a97_a1f92618d723.slice/crio-3f264160bd6f9db88b2228f4c9af3b5425685fc76b235ba69a58f8997f1527ca WatchSource:0}: Error finding container 3f264160bd6f9db88b2228f4c9af3b5425685fc76b235ba69a58f8997f1527ca: Status 404 returned error can't find the container with id 3f264160bd6f9db88b2228f4c9af3b5425685fc76b235ba69a58f8997f1527ca
Sep 14 10:54:18.704632 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:18.704612    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 10:54:19.288775 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:19.288742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk" event={"ID":"e7ff18e7-0c25-47e3-8a97-a1f92618d723","Type":"ContainerStarted","Data":"3f264160bd6f9db88b2228f4c9af3b5425685fc76b235ba69a58f8997f1527ca"}
Sep 14 10:54:25.316415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:25.316384    2561 generic.go:356] "Generic (PLEG): container finished" podID="e7ff18e7-0c25-47e3-8a97-a1f92618d723" containerID="e8e4367c5449c155c3dcb63eeb16749f050857a9eec1962cc32e258177f6bf06" exitCode=0
Sep 14 10:54:25.316907 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:25.316442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk" event={"ID":"e7ff18e7-0c25-47e3-8a97-a1f92618d723","Type":"ContainerDied","Data":"e8e4367c5449c155c3dcb63eeb16749f050857a9eec1962cc32e258177f6bf06"}
Sep 14 10:54:27.325805 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:27.325769    2561 generic.go:356] "Generic (PLEG): container finished" podID="e7ff18e7-0c25-47e3-8a97-a1f92618d723" containerID="a11657131d7c0e7185e9fd871d79bd173b2cfff8378e50c04bf026626f610f3e" exitCode=0
Sep 14 10:54:27.326082 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:27.325819    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk" event={"ID":"e7ff18e7-0c25-47e3-8a97-a1f92618d723","Type":"ContainerDied","Data":"a11657131d7c0e7185e9fd871d79bd173b2cfff8378e50c04bf026626f610f3e"}
Sep 14 10:54:34.356538 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:34.356503    2561 generic.go:356] "Generic (PLEG): container finished" podID="e7ff18e7-0c25-47e3-8a97-a1f92618d723" containerID="67abc555c9a43f9a138b5cf520329aa0f407be989235d255488f86179c396013" exitCode=0
Sep 14 10:54:34.356982 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:34.356559    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk" event={"ID":"e7ff18e7-0c25-47e3-8a97-a1f92618d723","Type":"ContainerDied","Data":"67abc555c9a43f9a138b5cf520329aa0f407be989235d255488f86179c396013"}
Sep 14 10:54:35.484596 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:35.484573    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"
Sep 14 10:54:35.655202 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:35.655124    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ff18e7-0c25-47e3-8a97-a1f92618d723-bundle\") pod \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\" (UID: \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\") "
Sep 14 10:54:35.655202 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:35.655198    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ff18e7-0c25-47e3-8a97-a1f92618d723-util\") pod \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\" (UID: \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\") "
Sep 14 10:54:35.655424 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:35.655228    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2qx7\" (UniqueName: \"kubernetes.io/projected/e7ff18e7-0c25-47e3-8a97-a1f92618d723-kube-api-access-l2qx7\") pod \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\" (UID: \"e7ff18e7-0c25-47e3-8a97-a1f92618d723\") "
Sep 14 10:54:35.655834 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:35.655809    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff18e7-0c25-47e3-8a97-a1f92618d723-bundle" (OuterVolumeSpecName: "bundle") pod "e7ff18e7-0c25-47e3-8a97-a1f92618d723" (UID: "e7ff18e7-0c25-47e3-8a97-a1f92618d723"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:54:35.657335 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:35.657314    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ff18e7-0c25-47e3-8a97-a1f92618d723-kube-api-access-l2qx7" (OuterVolumeSpecName: "kube-api-access-l2qx7") pod "e7ff18e7-0c25-47e3-8a97-a1f92618d723" (UID: "e7ff18e7-0c25-47e3-8a97-a1f92618d723"). InnerVolumeSpecName "kube-api-access-l2qx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:54:35.660940 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:35.660916    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff18e7-0c25-47e3-8a97-a1f92618d723-util" (OuterVolumeSpecName: "util") pod "e7ff18e7-0c25-47e3-8a97-a1f92618d723" (UID: "e7ff18e7-0c25-47e3-8a97-a1f92618d723"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:54:35.756292 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:35.756269    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ff18e7-0c25-47e3-8a97-a1f92618d723-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:54:35.756292 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:35.756290    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2qx7\" (UniqueName: \"kubernetes.io/projected/e7ff18e7-0c25-47e3-8a97-a1f92618d723-kube-api-access-l2qx7\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:54:35.756429 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:35.756300    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ff18e7-0c25-47e3-8a97-a1f92618d723-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:54:36.367095 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:36.367070    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk"
Sep 14 10:54:36.367095 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:36.367080    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhxhk" event={"ID":"e7ff18e7-0c25-47e3-8a97-a1f92618d723","Type":"ContainerDied","Data":"3f264160bd6f9db88b2228f4c9af3b5425685fc76b235ba69a58f8997f1527ca"}
Sep 14 10:54:36.367292 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:36.367118    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f264160bd6f9db88b2228f4c9af3b5425685fc76b235ba69a58f8997f1527ca"
Sep 14 10:54:42.342465 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.342427    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln"]
Sep 14 10:54:42.342934 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.342857    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ff18e7-0c25-47e3-8a97-a1f92618d723" containerName="pull"
Sep 14 10:54:42.342934 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.342875    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ff18e7-0c25-47e3-8a97-a1f92618d723" containerName="pull"
Sep 14 10:54:42.342934 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.342887    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ff18e7-0c25-47e3-8a97-a1f92618d723" containerName="extract"
Sep 14 10:54:42.342934 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.342894    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ff18e7-0c25-47e3-8a97-a1f92618d723" containerName="extract"
Sep 14 10:54:42.342934 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.342921    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ff18e7-0c25-47e3-8a97-a1f92618d723" containerName="util"
Sep 14 10:54:42.342934 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.342929    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ff18e7-0c25-47e3-8a97-a1f92618d723" containerName="util"
Sep 14 10:54:42.343217 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.343002    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7ff18e7-0c25-47e3-8a97-a1f92618d723" containerName="extract"
Sep 14 10:54:42.345499 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.345479    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln"
Sep 14 10:54:42.348149 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.348128    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-grc5t\""
Sep 14 10:54:42.350700 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.350665    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 14 10:54:42.350807 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.350676    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 14 10:54:42.402985 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.402961    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6d3a494-a38c-4554-ac6f-2bad33a0846b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-v85ln\" (UID: \"d6d3a494-a38c-4554-ac6f-2bad33a0846b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln"
Sep 14 10:54:42.403067 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.403007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clwkf\" (UniqueName: \"kubernetes.io/projected/d6d3a494-a38c-4554-ac6f-2bad33a0846b-kube-api-access-clwkf\") pod \"cert-manager-operator-controller-manager-55c444bf86-v85ln\" (UID: \"d6d3a494-a38c-4554-ac6f-2bad33a0846b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln"
Sep 14 10:54:42.503306 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.503286    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clwkf\" (UniqueName: \"kubernetes.io/projected/d6d3a494-a38c-4554-ac6f-2bad33a0846b-kube-api-access-clwkf\") pod \"cert-manager-operator-controller-manager-55c444bf86-v85ln\" (UID: \"d6d3a494-a38c-4554-ac6f-2bad33a0846b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln"
Sep 14 10:54:42.503419 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.503333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6d3a494-a38c-4554-ac6f-2bad33a0846b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-v85ln\" (UID: \"d6d3a494-a38c-4554-ac6f-2bad33a0846b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln"
Sep 14 10:54:42.503614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.503600    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6d3a494-a38c-4554-ac6f-2bad33a0846b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-v85ln\" (UID: \"d6d3a494-a38c-4554-ac6f-2bad33a0846b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln"
Sep 14 10:54:42.516775 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.516748    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clwkf\" (UniqueName: \"kubernetes.io/projected/d6d3a494-a38c-4554-ac6f-2bad33a0846b-kube-api-access-clwkf\") pod \"cert-manager-operator-controller-manager-55c444bf86-v85ln\" (UID: \"d6d3a494-a38c-4554-ac6f-2bad33a0846b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln"
Sep 14 10:54:42.554356 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.554331    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln"]
Sep 14 10:54:42.654032 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.654010    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln"
Sep 14 10:54:42.950683 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:42.950611    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln"]
Sep 14 10:54:42.958103 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:54:42.958073    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d3a494_a38c_4554_ac6f_2bad33a0846b.slice/crio-7af5792653e8e158d8a9f189e9c4286066c029a96bd5a4e216b3b6d50069785f WatchSource:0}: Error finding container 7af5792653e8e158d8a9f189e9c4286066c029a96bd5a4e216b3b6d50069785f: Status 404 returned error can't find the container with id 7af5792653e8e158d8a9f189e9c4286066c029a96bd5a4e216b3b6d50069785f
Sep 14 10:54:43.398623 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:43.398587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln" event={"ID":"d6d3a494-a38c-4554-ac6f-2bad33a0846b","Type":"ContainerStarted","Data":"7af5792653e8e158d8a9f189e9c4286066c029a96bd5a4e216b3b6d50069785f"}
Sep 14 10:54:45.409535 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:45.409500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln" event={"ID":"d6d3a494-a38c-4554-ac6f-2bad33a0846b","Type":"ContainerStarted","Data":"db97c8d7f44d2901d5d2825aaa545970b10cb2fccaddf31e9ca31e0878ce181a"}
Sep 14 10:54:45.480113 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:45.480064    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v85ln" podStartSLOduration=1.831337158 podStartE2EDuration="3.480049296s" totalImagesPullingTime="1.648712138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:54:45.478683315 +0000 UTC m=+390.163367704" watchObservedRunningTime="2026-09-14 10:54:45.480049296 +0000 UTC m=+390.164733684"
Sep 14 10:54:46.649922 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.649885    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"]
Sep 14 10:54:46.653122 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.653106    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"
Sep 14 10:54:46.659664 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.659556    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 10:54:46.659664 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.659559    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 10:54:46.660615 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.660595    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8p4wd\""
Sep 14 10:54:46.740280 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.740253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l\" (UID: \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"
Sep 14 10:54:46.740400 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.740293    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l\" (UID: \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"
Sep 14 10:54:46.740400 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.740319    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp52n\" (UniqueName: \"kubernetes.io/projected/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-kube-api-access-rp52n\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l\" (UID: \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"
Sep 14 10:54:46.766428 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.766402    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"]
Sep 14 10:54:46.841200 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.841175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l\" (UID: \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"
Sep 14 10:54:46.841329 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.841218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l\" (UID: \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"
Sep 14 10:54:46.841329 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.841238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp52n\" (UniqueName: \"kubernetes.io/projected/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-kube-api-access-rp52n\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l\" (UID: \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"
Sep 14 10:54:46.841529 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.841510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l\" (UID: \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"
Sep 14 10:54:46.841566 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.841546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l\" (UID: \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"
Sep 14 10:54:46.864387 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.864363    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp52n\" (UniqueName: \"kubernetes.io/projected/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-kube-api-access-rp52n\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l\" (UID: \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"
Sep 14 10:54:46.961855 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:46.961798    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"
Sep 14 10:54:47.342847 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:47.342818    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"]
Sep 14 10:54:47.345318 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:54:47.345292    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae05dcfe_1fec_4d54_afea_c4c5b0a6cd0f.slice/crio-c85110a7f267b88cc89fd39b6e2328d42bf743e203377e2db9106516608f7daa WatchSource:0}: Error finding container c85110a7f267b88cc89fd39b6e2328d42bf743e203377e2db9106516608f7daa: Status 404 returned error can't find the container with id c85110a7f267b88cc89fd39b6e2328d42bf743e203377e2db9106516608f7daa
Sep 14 10:54:47.419282 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:47.419245    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l" event={"ID":"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f","Type":"ContainerStarted","Data":"e6eee98bb79039919c97d4ec8f1940127a5d556fdf83f4190f0a8f671fbc254d"}
Sep 14 10:54:47.419415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:47.419299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l" event={"ID":"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f","Type":"ContainerStarted","Data":"c85110a7f267b88cc89fd39b6e2328d42bf743e203377e2db9106516608f7daa"}
Sep 14 10:54:48.424675 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.424639    2561 generic.go:356] "Generic (PLEG): container finished" podID="ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f" containerID="e6eee98bb79039919c97d4ec8f1940127a5d556fdf83f4190f0a8f671fbc254d" exitCode=0
Sep 14 10:54:48.425048 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.424748    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l" event={"ID":"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f","Type":"ContainerDied","Data":"e6eee98bb79039919c97d4ec8f1940127a5d556fdf83f4190f0a8f671fbc254d"}
Sep 14 10:54:48.477998 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.477969    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4lvlh"]
Sep 14 10:54:48.481409 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.481379    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4lvlh"
Sep 14 10:54:48.494555 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.494538    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 14 10:54:48.494693 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.494669    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 14 10:54:48.494799 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.494724    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ww9fh\""
Sep 14 10:54:48.507727 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.506264    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4lvlh"]
Sep 14 10:54:48.554378 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.554345    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv95z\" (UniqueName: \"kubernetes.io/projected/b7393b4b-262a-4dcb-87ec-e6e6d4603182-kube-api-access-vv95z\") pod \"cert-manager-webhook-6486ffd997-4lvlh\" (UID: \"b7393b4b-262a-4dcb-87ec-e6e6d4603182\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4lvlh"
Sep 14 10:54:48.554510 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.554405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7393b4b-262a-4dcb-87ec-e6e6d4603182-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4lvlh\" (UID: \"b7393b4b-262a-4dcb-87ec-e6e6d4603182\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4lvlh"
Sep 14 10:54:48.655648 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.655616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7393b4b-262a-4dcb-87ec-e6e6d4603182-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4lvlh\" (UID: \"b7393b4b-262a-4dcb-87ec-e6e6d4603182\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4lvlh"
Sep 14 10:54:48.655817 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.655669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv95z\" (UniqueName: \"kubernetes.io/projected/b7393b4b-262a-4dcb-87ec-e6e6d4603182-kube-api-access-vv95z\") pod \"cert-manager-webhook-6486ffd997-4lvlh\" (UID: \"b7393b4b-262a-4dcb-87ec-e6e6d4603182\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4lvlh"
Sep 14 10:54:48.684046 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.683980    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7393b4b-262a-4dcb-87ec-e6e6d4603182-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4lvlh\" (UID: \"b7393b4b-262a-4dcb-87ec-e6e6d4603182\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4lvlh"
Sep 14 10:54:48.685011 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.684995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv95z\" (UniqueName: \"kubernetes.io/projected/b7393b4b-262a-4dcb-87ec-e6e6d4603182-kube-api-access-vv95z\") pod \"cert-manager-webhook-6486ffd997-4lvlh\" (UID: \"b7393b4b-262a-4dcb-87ec-e6e6d4603182\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4lvlh"
Sep 14 10:54:48.803065 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.803040    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4lvlh"
Sep 14 10:54:48.886632 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.886604    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq"]
Sep 14 10:54:48.890181 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.890162    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq"
Sep 14 10:54:48.893449 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.893433    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mpb5g\""
Sep 14 10:54:48.914424 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.914400    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq"]
Sep 14 10:54:48.958117 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.958060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5tv9\" (UniqueName: \"kubernetes.io/projected/292f4543-7933-4f05-b619-e50429ec73ec-kube-api-access-k5tv9\") pod \"cert-manager-cainjector-559cdd8c87-tjqpq\" (UID: \"292f4543-7933-4f05-b619-e50429ec73ec\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq"
Sep 14 10:54:48.958199 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:48.958118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/292f4543-7933-4f05-b619-e50429ec73ec-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tjqpq\" (UID: \"292f4543-7933-4f05-b619-e50429ec73ec\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq"
Sep 14 10:54:49.058700 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:49.058677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5tv9\" (UniqueName: \"kubernetes.io/projected/292f4543-7933-4f05-b619-e50429ec73ec-kube-api-access-k5tv9\") pod \"cert-manager-cainjector-559cdd8c87-tjqpq\" (UID: \"292f4543-7933-4f05-b619-e50429ec73ec\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq"
Sep 14 10:54:49.058828 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:49.058799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/292f4543-7933-4f05-b619-e50429ec73ec-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tjqpq\" (UID: \"292f4543-7933-4f05-b619-e50429ec73ec\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq"
Sep 14 10:54:49.096359 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:49.096333    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4lvlh"]
Sep 14 10:54:49.098221 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:49.098194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/292f4543-7933-4f05-b619-e50429ec73ec-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tjqpq\" (UID: \"292f4543-7933-4f05-b619-e50429ec73ec\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq"
Sep 14 10:54:49.099661 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:54:49.099643    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7393b4b_262a_4dcb_87ec_e6e6d4603182.slice/crio-8be2991f102661ca35ad7439202e74ce3514dec9690016271ef310e57f7dc8b7 WatchSource:0}: Error finding container 8be2991f102661ca35ad7439202e74ce3514dec9690016271ef310e57f7dc8b7: Status 404 returned error can't find the container with id 8be2991f102661ca35ad7439202e74ce3514dec9690016271ef310e57f7dc8b7
Sep 14 10:54:49.129355 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:49.129328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5tv9\" (UniqueName: \"kubernetes.io/projected/292f4543-7933-4f05-b619-e50429ec73ec-kube-api-access-k5tv9\") pod \"cert-manager-cainjector-559cdd8c87-tjqpq\" (UID: \"292f4543-7933-4f05-b619-e50429ec73ec\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq"
Sep 14 10:54:49.199606 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:49.199584    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq"
Sep 14 10:54:49.430327 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:49.430295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4lvlh" event={"ID":"b7393b4b-262a-4dcb-87ec-e6e6d4603182","Type":"ContainerStarted","Data":"8be2991f102661ca35ad7439202e74ce3514dec9690016271ef310e57f7dc8b7"}
Sep 14 10:54:49.491791 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:49.491765    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq"]
Sep 14 10:54:49.493613 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:54:49.493584    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod292f4543_7933_4f05_b619_e50429ec73ec.slice/crio-6a2d82d49846c9cc272cb9d0f3e6c81689af5d5b25032fa8527c7ef953495ba4 WatchSource:0}: Error finding container 6a2d82d49846c9cc272cb9d0f3e6c81689af5d5b25032fa8527c7ef953495ba4: Status 404 returned error can't find the container with id 6a2d82d49846c9cc272cb9d0f3e6c81689af5d5b25032fa8527c7ef953495ba4
Sep 14 10:54:50.441498 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:50.441459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq" event={"ID":"292f4543-7933-4f05-b619-e50429ec73ec","Type":"ContainerStarted","Data":"6a2d82d49846c9cc272cb9d0f3e6c81689af5d5b25032fa8527c7ef953495ba4"}
Sep 14 10:54:52.453446 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:52.453404    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq" event={"ID":"292f4543-7933-4f05-b619-e50429ec73ec","Type":"ContainerStarted","Data":"0f0d8ac38c56f242f2ab48076ba9bfb29a5adf179a13297ec0eb04f505036180"}
Sep 14 10:54:52.455119 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:52.455090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4lvlh" event={"ID":"b7393b4b-262a-4dcb-87ec-e6e6d4603182","Type":"ContainerStarted","Data":"de01709c140e602163dcad1397f339f80b05c8f17904bb35ceaf286daa521af5"}
Sep 14 10:54:52.455260 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:52.455211    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4lvlh"
Sep 14 10:54:52.504136 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:52.504092    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tjqpq" podStartSLOduration=2.189594981 podStartE2EDuration="4.504078223s" totalImagesPullingTime="2.314483242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:54:52.502263192 +0000 UTC m=+397.186947580" watchObservedRunningTime="2026-09-14 10:54:52.504078223 +0000 UTC m=+397.188762612"
Sep 14 10:54:52.571455 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:52.571402    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-4lvlh" podStartSLOduration=1.869562977 podStartE2EDuration="4.571383195s" totalImagesPullingTime="2.701820218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:54:52.57064823 +0000 UTC m=+397.255332620" watchObservedRunningTime="2026-09-14 10:54:52.571383195 +0000 UTC m=+397.256067580"
Sep 14 10:54:58.462137 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:54:58.462108    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4lvlh"
Sep 14 10:55:11.537303 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:11.537276    2561 generic.go:356] "Generic (PLEG): container finished" podID="ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f" containerID="8ced4e3d7408ae24532ddbf8336412ea416fa92c3aeb3a8a7db6e86ab99e261c" exitCode=0
Sep 14 10:55:11.537587 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:11.537356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l" event={"ID":"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f","Type":"ContainerDied","Data":"8ced4e3d7408ae24532ddbf8336412ea416fa92c3aeb3a8a7db6e86ab99e261c"}
Sep 14 10:55:12.543856 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:12.543821    2561 generic.go:356] "Generic (PLEG): container finished" podID="ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f" containerID="4bd48da89ed99020945e4b318b3820538d69649ddb17504e7f9c6bc9dcde5cf7" exitCode=0
Sep 14 10:55:12.544255 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:12.543872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l" event={"ID":"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f","Type":"ContainerDied","Data":"4bd48da89ed99020945e4b318b3820538d69649ddb17504e7f9c6bc9dcde5cf7"}
Sep 14 10:55:13.667403 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:13.667380    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"
Sep 14 10:55:13.765422 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:13.765396    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp52n\" (UniqueName: \"kubernetes.io/projected/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-kube-api-access-rp52n\") pod \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\" (UID: \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\") "
Sep 14 10:55:13.765574 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:13.765429    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-util\") pod \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\" (UID: \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\") "
Sep 14 10:55:13.765574 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:13.765452    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-bundle\") pod \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\" (UID: \"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f\") "
Sep 14 10:55:13.765871 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:13.765848    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-bundle" (OuterVolumeSpecName: "bundle") pod "ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f" (UID: "ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:55:13.767485 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:13.767465    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-kube-api-access-rp52n" (OuterVolumeSpecName: "kube-api-access-rp52n") pod "ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f" (UID: "ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f"). InnerVolumeSpecName "kube-api-access-rp52n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:55:13.770084 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:13.770064    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-util" (OuterVolumeSpecName: "util") pod "ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f" (UID: "ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:55:13.866621 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:13.866595    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp52n\" (UniqueName: \"kubernetes.io/projected/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-kube-api-access-rp52n\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:55:13.866621 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:13.866617    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:55:13.866795 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:13.866628    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:55:14.555067 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:14.555033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l" event={"ID":"ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f","Type":"ContainerDied","Data":"c85110a7f267b88cc89fd39b6e2328d42bf743e203377e2db9106516608f7daa"}
Sep 14 10:55:14.555245 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:14.555078    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c85110a7f267b88cc89fd39b6e2328d42bf743e203377e2db9106516608f7daa"
Sep 14 10:55:14.555245 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:14.555050    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9sw2l"
Sep 14 10:55:18.678273 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.678240    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh"]
Sep 14 10:55:18.678656 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.678570    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f" containerName="extract"
Sep 14 10:55:18.678656 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.678582    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f" containerName="extract"
Sep 14 10:55:18.678656 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.678594    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f" containerName="pull"
Sep 14 10:55:18.678656 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.678602    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f" containerName="pull"
Sep 14 10:55:18.678656 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.678620    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f" containerName="util"
Sep 14 10:55:18.678656 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.678626    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f" containerName="util"
Sep 14 10:55:18.678886 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.678669    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae05dcfe-1fec-4d54-afea-c4c5b0a6cd0f" containerName="extract"
Sep 14 10:55:18.681720 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.681693    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh"
Sep 14 10:55:18.685134 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.685112    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 14 10:55:18.685263 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.685162    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 14 10:55:18.686234 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.686217    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-vh4mh\""
Sep 14 10:55:18.696477 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.696449    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh"]
Sep 14 10:55:18.802949 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.802919    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b8a38c7-6280-4985-8ab9-d8b527da060e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xg5xh\" (UID: \"6b8a38c7-6280-4985-8ab9-d8b527da060e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh"
Sep 14 10:55:18.803079 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.802951    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb65r\" (UniqueName: \"kubernetes.io/projected/6b8a38c7-6280-4985-8ab9-d8b527da060e-kube-api-access-qb65r\") pod \"openshift-lws-operator-fd8ccff4c-xg5xh\" (UID: \"6b8a38c7-6280-4985-8ab9-d8b527da060e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh"
Sep 14 10:55:18.903605 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.903567    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b8a38c7-6280-4985-8ab9-d8b527da060e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xg5xh\" (UID: \"6b8a38c7-6280-4985-8ab9-d8b527da060e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh"
Sep 14 10:55:18.903605 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.903609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb65r\" (UniqueName: \"kubernetes.io/projected/6b8a38c7-6280-4985-8ab9-d8b527da060e-kube-api-access-qb65r\") pod \"openshift-lws-operator-fd8ccff4c-xg5xh\" (UID: \"6b8a38c7-6280-4985-8ab9-d8b527da060e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh"
Sep 14 10:55:18.904038 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.904016    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b8a38c7-6280-4985-8ab9-d8b527da060e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xg5xh\" (UID: \"6b8a38c7-6280-4985-8ab9-d8b527da060e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh"
Sep 14 10:55:18.914578 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.914551    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb65r\" (UniqueName: \"kubernetes.io/projected/6b8a38c7-6280-4985-8ab9-d8b527da060e-kube-api-access-qb65r\") pod \"openshift-lws-operator-fd8ccff4c-xg5xh\" (UID: \"6b8a38c7-6280-4985-8ab9-d8b527da060e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh"
Sep 14 10:55:18.990468 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:18.990388    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh"
Sep 14 10:55:19.265852 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:19.265830    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh"]
Sep 14 10:55:19.267819 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:55:19.267792    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b8a38c7_6280_4985_8ab9_d8b527da060e.slice/crio-24818c5140a108fe14fb016ea6a8f19effac25dbdf47bf2125de4091ec4ceb2a WatchSource:0}: Error finding container 24818c5140a108fe14fb016ea6a8f19effac25dbdf47bf2125de4091ec4ceb2a: Status 404 returned error can't find the container with id 24818c5140a108fe14fb016ea6a8f19effac25dbdf47bf2125de4091ec4ceb2a
Sep 14 10:55:19.577273 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:19.577196    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh" event={"ID":"6b8a38c7-6280-4985-8ab9-d8b527da060e","Type":"ContainerStarted","Data":"24818c5140a108fe14fb016ea6a8f19effac25dbdf47bf2125de4091ec4ceb2a"}
Sep 14 10:55:21.588808 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:21.588775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh" event={"ID":"6b8a38c7-6280-4985-8ab9-d8b527da060e","Type":"ContainerStarted","Data":"5cbc31240fb3c179fa3ff439d5d91bd220243c9d8e8db08093aff2e4e55e1867"}
Sep 14 10:55:21.607912 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:21.607865    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg5xh" podStartSLOduration=1.8146252600000001 podStartE2EDuration="3.607849051s" totalImagesPullingTime="1.793223791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:55:21.606880195 +0000 UTC m=+426.291564580" watchObservedRunningTime="2026-09-14 10:55:21.607849051 +0000 UTC m=+426.292533440"
Sep 14 10:55:23.664669 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.664639    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66b78f9596-w8tbf"]
Sep 14 10:55:23.667935 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.667915    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.699400 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.699370    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b78f9596-w8tbf"]
Sep 14 10:55:23.742370 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.742345    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfba9294-743c-4057-a30e-f51366280342-console-serving-cert\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.742468 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.742375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfba9294-743c-4057-a30e-f51366280342-trusted-ca-bundle\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.742468 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.742410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfba9294-743c-4057-a30e-f51366280342-console-oauth-config\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.742543 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.742460    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfba9294-743c-4057-a30e-f51366280342-console-config\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.742543 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.742495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfba9294-743c-4057-a30e-f51366280342-service-ca\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.742543 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.742511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfba9294-743c-4057-a30e-f51366280342-oauth-serving-cert\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.742543 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.742539    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp9wr\" (UniqueName: \"kubernetes.io/projected/bfba9294-743c-4057-a30e-f51366280342-kube-api-access-fp9wr\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.843853 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.843820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfba9294-743c-4057-a30e-f51366280342-console-serving-cert\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.843853 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.843854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfba9294-743c-4057-a30e-f51366280342-trusted-ca-bundle\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.844055 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.843880    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfba9294-743c-4057-a30e-f51366280342-console-oauth-config\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.844055 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.843898    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfba9294-743c-4057-a30e-f51366280342-console-config\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.844055 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.843925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfba9294-743c-4057-a30e-f51366280342-service-ca\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.844055 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.843946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfba9294-743c-4057-a30e-f51366280342-oauth-serving-cert\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.844055 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.843988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp9wr\" (UniqueName: \"kubernetes.io/projected/bfba9294-743c-4057-a30e-f51366280342-kube-api-access-fp9wr\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.844791 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.844761    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfba9294-743c-4057-a30e-f51366280342-service-ca\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.844888 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.844767    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfba9294-743c-4057-a30e-f51366280342-trusted-ca-bundle\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.844888 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.844868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfba9294-743c-4057-a30e-f51366280342-console-config\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.844888 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.844871    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfba9294-743c-4057-a30e-f51366280342-oauth-serving-cert\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.846250 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.846229    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfba9294-743c-4057-a30e-f51366280342-console-oauth-config\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.846393 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.846376    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfba9294-743c-4057-a30e-f51366280342-console-serving-cert\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.854381 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.854353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp9wr\" (UniqueName: \"kubernetes.io/projected/bfba9294-743c-4057-a30e-f51366280342-kube-api-access-fp9wr\") pod \"console-66b78f9596-w8tbf\" (UID: \"bfba9294-743c-4057-a30e-f51366280342\") " pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:23.977984 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:23.977899    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:24.274293 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.274268    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b78f9596-w8tbf"]
Sep 14 10:55:24.276623 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:55:24.276601    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfba9294_743c_4057_a30e_f51366280342.slice/crio-218fa26018efb945b756f43ccbe2295c20578efdbb528de8d37999a165f06505 WatchSource:0}: Error finding container 218fa26018efb945b756f43ccbe2295c20578efdbb528de8d37999a165f06505: Status 404 returned error can't find the container with id 218fa26018efb945b756f43ccbe2295c20578efdbb528de8d37999a165f06505
Sep 14 10:55:24.327261 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.327239    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"]
Sep 14 10:55:24.331683 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.331665    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"
Sep 14 10:55:24.337126 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.337104    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8p4wd\""
Sep 14 10:55:24.337271 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.337252    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 10:55:24.337783 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.337765    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 10:55:24.360929 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.360906    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"]
Sep 14 10:55:24.450070 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.450036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4\" (UID: \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"
Sep 14 10:55:24.450070 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.450072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4\" (UID: \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"
Sep 14 10:55:24.450299 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.450094    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7s8d\" (UniqueName: \"kubernetes.io/projected/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-kube-api-access-b7s8d\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4\" (UID: \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"
Sep 14 10:55:24.551475 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.551407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4\" (UID: \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"
Sep 14 10:55:24.551475 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.551442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4\" (UID: \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"
Sep 14 10:55:24.551475 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.551461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7s8d\" (UniqueName: \"kubernetes.io/projected/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-kube-api-access-b7s8d\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4\" (UID: \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"
Sep 14 10:55:24.551841 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.551819    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4\" (UID: \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"
Sep 14 10:55:24.551883 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.551842    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4\" (UID: \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"
Sep 14 10:55:24.564022 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.564000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7s8d\" (UniqueName: \"kubernetes.io/projected/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-kube-api-access-b7s8d\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4\" (UID: \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"
Sep 14 10:55:24.604264 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.604237    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b78f9596-w8tbf" event={"ID":"bfba9294-743c-4057-a30e-f51366280342","Type":"ContainerStarted","Data":"7ec5e0951c4083e0b50d3d964f27d3ee29611a26378c7275bdc6666cb381db00"}
Sep 14 10:55:24.604361 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.604280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b78f9596-w8tbf" event={"ID":"bfba9294-743c-4057-a30e-f51366280342","Type":"ContainerStarted","Data":"218fa26018efb945b756f43ccbe2295c20578efdbb528de8d37999a165f06505"}
Sep 14 10:55:24.640250 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.640231    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"
Sep 14 10:55:24.649516 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.649479    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66b78f9596-w8tbf" podStartSLOduration=1.6494660049999998 podStartE2EDuration="1.649466005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:55:24.640265757 +0000 UTC m=+429.324950145" watchObservedRunningTime="2026-09-14 10:55:24.649466005 +0000 UTC m=+429.334150442"
Sep 14 10:55:24.950525 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:24.950500    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"]
Sep 14 10:55:24.952877 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:55:24.952848    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e31357_489d_4bed_aa8c_b05dd0a99bd3.slice/crio-2994c9d321b5b68cee48667ae78d8c110e656d83695756c37bd37c3ffb5d312a WatchSource:0}: Error finding container 2994c9d321b5b68cee48667ae78d8c110e656d83695756c37bd37c3ffb5d312a: Status 404 returned error can't find the container with id 2994c9d321b5b68cee48667ae78d8c110e656d83695756c37bd37c3ffb5d312a
Sep 14 10:55:25.610193 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:25.610161    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4e31357-489d-4bed-aa8c-b05dd0a99bd3" containerID="95a2def9e9362da317c559054ebf0d76a5f17c03ee807783f9c45c872611e1f4" exitCode=0
Sep 14 10:55:25.610359 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:25.610230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4" event={"ID":"d4e31357-489d-4bed-aa8c-b05dd0a99bd3","Type":"ContainerDied","Data":"95a2def9e9362da317c559054ebf0d76a5f17c03ee807783f9c45c872611e1f4"}
Sep 14 10:55:25.610359 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:25.610271    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4" event={"ID":"d4e31357-489d-4bed-aa8c-b05dd0a99bd3","Type":"ContainerStarted","Data":"2994c9d321b5b68cee48667ae78d8c110e656d83695756c37bd37c3ffb5d312a"}
Sep 14 10:55:26.616904 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:26.616876    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4e31357-489d-4bed-aa8c-b05dd0a99bd3" containerID="ebb2da040bdc30ae0b1886b5a2fad9bcd0baf32db35701a51334d102cc2b2e22" exitCode=0
Sep 14 10:55:26.617288 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:26.616963    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4" event={"ID":"d4e31357-489d-4bed-aa8c-b05dd0a99bd3","Type":"ContainerDied","Data":"ebb2da040bdc30ae0b1886b5a2fad9bcd0baf32db35701a51334d102cc2b2e22"}
Sep 14 10:55:27.623441 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:27.623412    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4e31357-489d-4bed-aa8c-b05dd0a99bd3" containerID="e316d4283ca7b4c320d5c0673d2672ae530d09d56070169ad3c7f65fc8c299f0" exitCode=0
Sep 14 10:55:27.623844 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:27.623500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4" event={"ID":"d4e31357-489d-4bed-aa8c-b05dd0a99bd3","Type":"ContainerDied","Data":"e316d4283ca7b4c320d5c0673d2672ae530d09d56070169ad3c7f65fc8c299f0"}
Sep 14 10:55:28.750508 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:28.750485    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"
Sep 14 10:55:28.886024 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:28.885955    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-util\") pod \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\" (UID: \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\") "
Sep 14 10:55:28.886024 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:28.886005    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-bundle\") pod \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\" (UID: \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\") "
Sep 14 10:55:28.886215 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:28.886078    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7s8d\" (UniqueName: \"kubernetes.io/projected/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-kube-api-access-b7s8d\") pod \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\" (UID: \"d4e31357-489d-4bed-aa8c-b05dd0a99bd3\") "
Sep 14 10:55:28.886860 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:28.886830    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-bundle" (OuterVolumeSpecName: "bundle") pod "d4e31357-489d-4bed-aa8c-b05dd0a99bd3" (UID: "d4e31357-489d-4bed-aa8c-b05dd0a99bd3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:55:28.888152 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:28.888131    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-kube-api-access-b7s8d" (OuterVolumeSpecName: "kube-api-access-b7s8d") pod "d4e31357-489d-4bed-aa8c-b05dd0a99bd3" (UID: "d4e31357-489d-4bed-aa8c-b05dd0a99bd3"). InnerVolumeSpecName "kube-api-access-b7s8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:55:28.891648 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:28.891627    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-util" (OuterVolumeSpecName: "util") pod "d4e31357-489d-4bed-aa8c-b05dd0a99bd3" (UID: "d4e31357-489d-4bed-aa8c-b05dd0a99bd3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:55:28.986984 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:28.986954    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7s8d\" (UniqueName: \"kubernetes.io/projected/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-kube-api-access-b7s8d\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:55:28.986984 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:28.986981    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:55:28.987129 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:28.986994    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e31357-489d-4bed-aa8c-b05dd0a99bd3-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:55:29.634486 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:29.634453    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4"
Sep 14 10:55:29.634649 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:29.634484    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6cps4" event={"ID":"d4e31357-489d-4bed-aa8c-b05dd0a99bd3","Type":"ContainerDied","Data":"2994c9d321b5b68cee48667ae78d8c110e656d83695756c37bd37c3ffb5d312a"}
Sep 14 10:55:29.634649 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:29.634528    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2994c9d321b5b68cee48667ae78d8c110e656d83695756c37bd37c3ffb5d312a"
Sep 14 10:55:33.978333 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:33.978288    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:33.978844 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:33.978358    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:33.983521 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:33.983498    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:34.660062 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:34.660034    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66b78f9596-w8tbf"
Sep 14 10:55:34.762475 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:34.762445    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-748f74fdbf-59rdf"]
Sep 14 10:55:35.302250 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.302220    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"]
Sep 14 10:55:35.302597 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.302548    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e31357-489d-4bed-aa8c-b05dd0a99bd3" containerName="util"
Sep 14 10:55:35.302597 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.302560    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e31357-489d-4bed-aa8c-b05dd0a99bd3" containerName="util"
Sep 14 10:55:35.302597 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.302570    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e31357-489d-4bed-aa8c-b05dd0a99bd3" containerName="extract"
Sep 14 10:55:35.302597 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.302575    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e31357-489d-4bed-aa8c-b05dd0a99bd3" containerName="extract"
Sep 14 10:55:35.302597 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.302590    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e31357-489d-4bed-aa8c-b05dd0a99bd3" containerName="pull"
Sep 14 10:55:35.302597 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.302596    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e31357-489d-4bed-aa8c-b05dd0a99bd3" containerName="pull"
Sep 14 10:55:35.302840 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.302643    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e31357-489d-4bed-aa8c-b05dd0a99bd3" containerName="extract"
Sep 14 10:55:35.306022 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.306007    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"
Sep 14 10:55:35.310317 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.310297    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 10:55:35.310435 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.310352    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8p4wd\""
Sep 14 10:55:35.312115 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.312092    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 10:55:35.332346 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.332326    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"]
Sep 14 10:55:35.435767 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.435732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e79e85-d179-4046-ab06-917443763e29-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22\" (UID: \"d9e79e85-d179-4046-ab06-917443763e29\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"
Sep 14 10:55:35.435913 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.435787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e79e85-d179-4046-ab06-917443763e29-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22\" (UID: \"d9e79e85-d179-4046-ab06-917443763e29\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"
Sep 14 10:55:35.435913 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.435826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppjff\" (UniqueName: \"kubernetes.io/projected/d9e79e85-d179-4046-ab06-917443763e29-kube-api-access-ppjff\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22\" (UID: \"d9e79e85-d179-4046-ab06-917443763e29\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"
Sep 14 10:55:35.536415 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.536384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e79e85-d179-4046-ab06-917443763e29-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22\" (UID: \"d9e79e85-d179-4046-ab06-917443763e29\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"
Sep 14 10:55:35.536592 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.536431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e79e85-d179-4046-ab06-917443763e29-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22\" (UID: \"d9e79e85-d179-4046-ab06-917443763e29\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"
Sep 14 10:55:35.536592 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.536531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppjff\" (UniqueName: \"kubernetes.io/projected/d9e79e85-d179-4046-ab06-917443763e29-kube-api-access-ppjff\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22\" (UID: \"d9e79e85-d179-4046-ab06-917443763e29\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"
Sep 14 10:55:35.536853 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.536832    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e79e85-d179-4046-ab06-917443763e29-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22\" (UID: \"d9e79e85-d179-4046-ab06-917443763e29\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"
Sep 14 10:55:35.536853 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.536844    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e79e85-d179-4046-ab06-917443763e29-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22\" (UID: \"d9e79e85-d179-4046-ab06-917443763e29\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"
Sep 14 10:55:35.567920 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.567863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppjff\" (UniqueName: \"kubernetes.io/projected/d9e79e85-d179-4046-ab06-917443763e29-kube-api-access-ppjff\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22\" (UID: \"d9e79e85-d179-4046-ab06-917443763e29\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"
Sep 14 10:55:35.614706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.614680    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"
Sep 14 10:55:35.913625 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:35.913604    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"]
Sep 14 10:55:35.915392 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:55:35.915369    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e79e85_d179_4046_ab06_917443763e29.slice/crio-4074d7f6f03db893fe3006498a558ea697b689cb8d593923e11474eb5cfe3077 WatchSource:0}: Error finding container 4074d7f6f03db893fe3006498a558ea697b689cb8d593923e11474eb5cfe3077: Status 404 returned error can't find the container with id 4074d7f6f03db893fe3006498a558ea697b689cb8d593923e11474eb5cfe3077
Sep 14 10:55:36.666802 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:36.666775    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9e79e85-d179-4046-ab06-917443763e29" containerID="20f70fbc0be7393da3848251eb020de84a41fe34f4f029a4e2f0b0f3ca44f773" exitCode=0
Sep 14 10:55:36.667152 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:36.666821    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22" event={"ID":"d9e79e85-d179-4046-ab06-917443763e29","Type":"ContainerDied","Data":"20f70fbc0be7393da3848251eb020de84a41fe34f4f029a4e2f0b0f3ca44f773"}
Sep 14 10:55:36.667152 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:36.666855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22" event={"ID":"d9e79e85-d179-4046-ab06-917443763e29","Type":"ContainerStarted","Data":"4074d7f6f03db893fe3006498a558ea697b689cb8d593923e11474eb5cfe3077"}
Sep 14 10:55:37.025648 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.025611    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"]
Sep 14 10:55:37.028940 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.028924    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.051954 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.051929    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 14 10:55:37.053140 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.053119    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tstsr\""
Sep 14 10:55:37.053243 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.053227    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 14 10:55:37.053298 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.053243    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 14 10:55:37.064295 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.064269    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"]
Sep 14 10:55:37.151574 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.151532    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/950120ed-49ef-4418-9409-860db1753d4b-cert\") pod \"lws-controller-manager-587bf9cb9-qsxsf\" (UID: \"950120ed-49ef-4418-9409-860db1753d4b\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.151752 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.151585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hflnm\" (UniqueName: \"kubernetes.io/projected/950120ed-49ef-4418-9409-860db1753d4b-kube-api-access-hflnm\") pod \"lws-controller-manager-587bf9cb9-qsxsf\" (UID: \"950120ed-49ef-4418-9409-860db1753d4b\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.151752 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.151706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/950120ed-49ef-4418-9409-860db1753d4b-manager-config\") pod \"lws-controller-manager-587bf9cb9-qsxsf\" (UID: \"950120ed-49ef-4418-9409-860db1753d4b\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.151862 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.151769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/950120ed-49ef-4418-9409-860db1753d4b-metrics-cert\") pod \"lws-controller-manager-587bf9cb9-qsxsf\" (UID: \"950120ed-49ef-4418-9409-860db1753d4b\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.252502 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.252468    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/950120ed-49ef-4418-9409-860db1753d4b-cert\") pod \"lws-controller-manager-587bf9cb9-qsxsf\" (UID: \"950120ed-49ef-4418-9409-860db1753d4b\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.252669 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.252508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hflnm\" (UniqueName: \"kubernetes.io/projected/950120ed-49ef-4418-9409-860db1753d4b-kube-api-access-hflnm\") pod \"lws-controller-manager-587bf9cb9-qsxsf\" (UID: \"950120ed-49ef-4418-9409-860db1753d4b\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.252669 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.252540    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/950120ed-49ef-4418-9409-860db1753d4b-manager-config\") pod \"lws-controller-manager-587bf9cb9-qsxsf\" (UID: \"950120ed-49ef-4418-9409-860db1753d4b\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.252669 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.252560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/950120ed-49ef-4418-9409-860db1753d4b-metrics-cert\") pod \"lws-controller-manager-587bf9cb9-qsxsf\" (UID: \"950120ed-49ef-4418-9409-860db1753d4b\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.253295 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.253273    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/950120ed-49ef-4418-9409-860db1753d4b-manager-config\") pod \"lws-controller-manager-587bf9cb9-qsxsf\" (UID: \"950120ed-49ef-4418-9409-860db1753d4b\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.254942 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.254916    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/950120ed-49ef-4418-9409-860db1753d4b-metrics-cert\") pod \"lws-controller-manager-587bf9cb9-qsxsf\" (UID: \"950120ed-49ef-4418-9409-860db1753d4b\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.255062 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.255006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/950120ed-49ef-4418-9409-860db1753d4b-cert\") pod \"lws-controller-manager-587bf9cb9-qsxsf\" (UID: \"950120ed-49ef-4418-9409-860db1753d4b\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.263480 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.263459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hflnm\" (UniqueName: \"kubernetes.io/projected/950120ed-49ef-4418-9409-860db1753d4b-kube-api-access-hflnm\") pod \"lws-controller-manager-587bf9cb9-qsxsf\" (UID: \"950120ed-49ef-4418-9409-860db1753d4b\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.337854 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.337834    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:37.673619 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.673584    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9e79e85-d179-4046-ab06-917443763e29" containerID="14ee54563601f15097cb43518a2b4f44b4187536e77b612c906149c514ffa3a4" exitCode=0
Sep 14 10:55:37.674019 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.673641    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22" event={"ID":"d9e79e85-d179-4046-ab06-917443763e29","Type":"ContainerDied","Data":"14ee54563601f15097cb43518a2b4f44b4187536e77b612c906149c514ffa3a4"}
Sep 14 10:55:37.679817 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.679797    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"]
Sep 14 10:55:37.680270 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:55:37.680244    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950120ed_49ef_4418_9409_860db1753d4b.slice/crio-80d753adcb4d2803bd4d3f922607ae33685b9a50073f652379b6fd96365b7503 WatchSource:0}: Error finding container 80d753adcb4d2803bd4d3f922607ae33685b9a50073f652379b6fd96365b7503: Status 404 returned error can't find the container with id 80d753adcb4d2803bd4d3f922607ae33685b9a50073f652379b6fd96365b7503
Sep 14 10:55:37.866468 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.866443    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"]
Sep 14 10:55:37.869724 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.869690    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:37.879464 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.879442    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rs8sp\""
Sep 14 10:55:37.879553 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.879524    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 14 10:55:37.879553 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.879546    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 14 10:55:37.880481 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.880469    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 14 10:55:37.880603 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.880590    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 14 10:55:37.914605 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.914584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"]
Sep 14 10:55:37.957915 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.957886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbnzf\" (UniqueName: \"kubernetes.io/projected/23430bd4-dfe5-440b-8f88-931054b06e46-kube-api-access-jbnzf\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:37.958044 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.957936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23430bd4-dfe5-440b-8f88-931054b06e46-tmp\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:37.958044 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.957963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/23430bd4-dfe5-440b-8f88-931054b06e46-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:37.958044 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.957987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23430bd4-dfe5-440b-8f88-931054b06e46-webhook-cert\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:37.958044 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:37.958021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/23430bd4-dfe5-440b-8f88-931054b06e46-manifests\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:38.059356 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.059288    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/23430bd4-dfe5-440b-8f88-931054b06e46-manifests\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:38.059482 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.059356    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbnzf\" (UniqueName: \"kubernetes.io/projected/23430bd4-dfe5-440b-8f88-931054b06e46-kube-api-access-jbnzf\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:38.059482 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.059388    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23430bd4-dfe5-440b-8f88-931054b06e46-tmp\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:38.059482 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.059405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/23430bd4-dfe5-440b-8f88-931054b06e46-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:38.059482 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.059420    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23430bd4-dfe5-440b-8f88-931054b06e46-webhook-cert\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:38.059803 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.059766    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/23430bd4-dfe5-440b-8f88-931054b06e46-manifests\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:38.061872 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.061851    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23430bd4-dfe5-440b-8f88-931054b06e46-tmp\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:38.062110 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.062088    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23430bd4-dfe5-440b-8f88-931054b06e46-webhook-cert\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:38.062606 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.062583    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/23430bd4-dfe5-440b-8f88-931054b06e46-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:38.069520 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.069498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbnzf\" (UniqueName: \"kubernetes.io/projected/23430bd4-dfe5-440b-8f88-931054b06e46-kube-api-access-jbnzf\") pod \"opendatahub-operator-controller-manager-f6d976668-9bwzw\" (UID: \"23430bd4-dfe5-440b-8f88-931054b06e46\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:38.211437 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.211409    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:38.542818 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:55:38.542788    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23430bd4_dfe5_440b_8f88_931054b06e46.slice/crio-f7df935729be440cef40728e9c4608a52b077ed7640fa2252bff65a5ff96ef3f WatchSource:0}: Error finding container f7df935729be440cef40728e9c4608a52b077ed7640fa2252bff65a5ff96ef3f: Status 404 returned error can't find the container with id f7df935729be440cef40728e9c4608a52b077ed7640fa2252bff65a5ff96ef3f
Sep 14 10:55:38.543255 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.543180    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"]
Sep 14 10:55:38.680926 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.680892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf" event={"ID":"950120ed-49ef-4418-9409-860db1753d4b","Type":"ContainerStarted","Data":"80d753adcb4d2803bd4d3f922607ae33685b9a50073f652379b6fd96365b7503"}
Sep 14 10:55:38.683560 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.683491    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9e79e85-d179-4046-ab06-917443763e29" containerID="f9903243efdff82fed21d7a62f2ab756c9c9031569420bd29849eaba3f068026" exitCode=0
Sep 14 10:55:38.683689 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.683577    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22" event={"ID":"d9e79e85-d179-4046-ab06-917443763e29","Type":"ContainerDied","Data":"f9903243efdff82fed21d7a62f2ab756c9c9031569420bd29849eaba3f068026"}
Sep 14 10:55:38.685254 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:38.685229    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw" event={"ID":"23430bd4-dfe5-440b-8f88-931054b06e46","Type":"ContainerStarted","Data":"f7df935729be440cef40728e9c4608a52b077ed7640fa2252bff65a5ff96ef3f"}
Sep 14 10:55:39.693873 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:39.693508    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf" event={"ID":"950120ed-49ef-4418-9409-860db1753d4b","Type":"ContainerStarted","Data":"5d6b270cd88bc279af3ac9a2ca25d8588bf25ea221c51ef34d13c6604c32f429"}
Sep 14 10:55:39.693873 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:39.693824    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:39.732389 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:39.732332    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf" podStartSLOduration=2.174179234 podStartE2EDuration="3.732309442s" totalImagesPullingTime="1.558130208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:55:39.729931768 +0000 UTC m=+444.414616157" watchObservedRunningTime="2026-09-14 10:55:39.732309442 +0000 UTC m=+444.416993832"
Sep 14 10:55:39.855936 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:39.855911    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"
Sep 14 10:55:39.978860 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:39.978777    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e79e85-d179-4046-ab06-917443763e29-util\") pod \"d9e79e85-d179-4046-ab06-917443763e29\" (UID: \"d9e79e85-d179-4046-ab06-917443763e29\") "
Sep 14 10:55:39.978860 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:39.978825    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e79e85-d179-4046-ab06-917443763e29-bundle\") pod \"d9e79e85-d179-4046-ab06-917443763e29\" (UID: \"d9e79e85-d179-4046-ab06-917443763e29\") "
Sep 14 10:55:39.979074 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:39.978878    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppjff\" (UniqueName: \"kubernetes.io/projected/d9e79e85-d179-4046-ab06-917443763e29-kube-api-access-ppjff\") pod \"d9e79e85-d179-4046-ab06-917443763e29\" (UID: \"d9e79e85-d179-4046-ab06-917443763e29\") "
Sep 14 10:55:39.979990 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:39.979802    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e79e85-d179-4046-ab06-917443763e29-bundle" (OuterVolumeSpecName: "bundle") pod "d9e79e85-d179-4046-ab06-917443763e29" (UID: "d9e79e85-d179-4046-ab06-917443763e29"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:55:39.982583 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:39.982551    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e79e85-d179-4046-ab06-917443763e29-kube-api-access-ppjff" (OuterVolumeSpecName: "kube-api-access-ppjff") pod "d9e79e85-d179-4046-ab06-917443763e29" (UID: "d9e79e85-d179-4046-ab06-917443763e29"). InnerVolumeSpecName "kube-api-access-ppjff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:55:39.985807 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:39.985758    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e79e85-d179-4046-ab06-917443763e29-util" (OuterVolumeSpecName: "util") pod "d9e79e85-d179-4046-ab06-917443763e29" (UID: "d9e79e85-d179-4046-ab06-917443763e29"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:55:40.079965 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:40.079923    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e79e85-d179-4046-ab06-917443763e29-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:55:40.079965 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:40.079960    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e79e85-d179-4046-ab06-917443763e29-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:55:40.080220 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:40.080013    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppjff\" (UniqueName: \"kubernetes.io/projected/d9e79e85-d179-4046-ab06-917443763e29-kube-api-access-ppjff\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:55:40.699775 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:40.699743    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22"
Sep 14 10:55:40.699775 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:40.699747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0g8j22" event={"ID":"d9e79e85-d179-4046-ab06-917443763e29","Type":"ContainerDied","Data":"4074d7f6f03db893fe3006498a558ea697b689cb8d593923e11474eb5cfe3077"}
Sep 14 10:55:40.700337 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:40.699841    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4074d7f6f03db893fe3006498a558ea697b689cb8d593923e11474eb5cfe3077"
Sep 14 10:55:42.711150 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:42.711119    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw" event={"ID":"23430bd4-dfe5-440b-8f88-931054b06e46","Type":"ContainerStarted","Data":"25daf7f8289bea5d13e127ef28f1617cc359b5cc45d8f37bd6ffeee9ea6cc1a3"}
Sep 14 10:55:43.717283 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:43.717247    2561 generic.go:356] "Generic (PLEG): container finished" podID="23430bd4-dfe5-440b-8f88-931054b06e46" containerID="25daf7f8289bea5d13e127ef28f1617cc359b5cc45d8f37bd6ffeee9ea6cc1a3" exitCode=0
Sep 14 10:55:43.717799 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:43.717296    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw" event={"ID":"23430bd4-dfe5-440b-8f88-931054b06e46","Type":"ContainerDied","Data":"25daf7f8289bea5d13e127ef28f1617cc359b5cc45d8f37bd6ffeee9ea6cc1a3"}
Sep 14 10:55:44.724837 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:44.724801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw" event={"ID":"23430bd4-dfe5-440b-8f88-931054b06e46","Type":"ContainerStarted","Data":"9f860eccc80d7662abaa7f9ea2a628234012706376aadd09dc574b55b1ed16d5"}
Sep 14 10:55:44.725234 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:44.724879    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:44.760570 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:44.760523    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw" podStartSLOduration=3.262760947 podStartE2EDuration="7.760510489s" totalImagesPullingTime="3.46784236s" totalInitContainerRuntime="1.029907182s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:55:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:55:44.758152212 +0000 UTC m=+449.442836601" watchObservedRunningTime="2026-09-14 10:55:44.760510489 +0000 UTC m=+449.445194874"
Sep 14 10:55:50.702767 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:50.702735    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-qsxsf"
Sep 14 10:55:55.732165 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:55.732130    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-9bwzw"
Sep 14 10:55:59.782067 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:55:59.782018    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-748f74fdbf-59rdf" podUID="acedcbd0-81ff-494e-ab84-de490f2e31d0" containerName="console" containerID="cri-o://a2ce1b287a06c52724943caee326e79645777ad5f0b9b7aa2ca8dbbca0efbfee" gracePeriod=15
Sep 14 10:56:00.029918 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.029891    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:56:00.142015 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.141987    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-serving-cert\") pod \"acedcbd0-81ff-494e-ab84-de490f2e31d0\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") "
Sep 14 10:56:00.142179 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.142024    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-config\") pod \"acedcbd0-81ff-494e-ab84-de490f2e31d0\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") "
Sep 14 10:56:00.142179 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.142075    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8kww\" (UniqueName: \"kubernetes.io/projected/acedcbd0-81ff-494e-ab84-de490f2e31d0-kube-api-access-r8kww\") pod \"acedcbd0-81ff-494e-ab84-de490f2e31d0\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") "
Sep 14 10:56:00.142179 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.142092    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-service-ca\") pod \"acedcbd0-81ff-494e-ab84-de490f2e31d0\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") "
Sep 14 10:56:00.142179 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.142140    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-oauth-serving-cert\") pod \"acedcbd0-81ff-494e-ab84-de490f2e31d0\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") "
Sep 14 10:56:00.142179 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.142160    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-trusted-ca-bundle\") pod \"acedcbd0-81ff-494e-ab84-de490f2e31d0\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") "
Sep 14 10:56:00.142440 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.142314    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-oauth-config\") pod \"acedcbd0-81ff-494e-ab84-de490f2e31d0\" (UID: \"acedcbd0-81ff-494e-ab84-de490f2e31d0\") "
Sep 14 10:56:00.142597 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.142564    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "acedcbd0-81ff-494e-ab84-de490f2e31d0" (UID: "acedcbd0-81ff-494e-ab84-de490f2e31d0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:56:00.142597 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.142580    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-service-ca" (OuterVolumeSpecName: "service-ca") pod "acedcbd0-81ff-494e-ab84-de490f2e31d0" (UID: "acedcbd0-81ff-494e-ab84-de490f2e31d0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:56:00.142946 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.142842    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "acedcbd0-81ff-494e-ab84-de490f2e31d0" (UID: "acedcbd0-81ff-494e-ab84-de490f2e31d0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:56:00.143040 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.142983    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-config" (OuterVolumeSpecName: "console-config") pod "acedcbd0-81ff-494e-ab84-de490f2e31d0" (UID: "acedcbd0-81ff-494e-ab84-de490f2e31d0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:56:00.144254 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.144228    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "acedcbd0-81ff-494e-ab84-de490f2e31d0" (UID: "acedcbd0-81ff-494e-ab84-de490f2e31d0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:56:00.144391 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.144369    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "acedcbd0-81ff-494e-ab84-de490f2e31d0" (UID: "acedcbd0-81ff-494e-ab84-de490f2e31d0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:56:00.144391 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.144382    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acedcbd0-81ff-494e-ab84-de490f2e31d0-kube-api-access-r8kww" (OuterVolumeSpecName: "kube-api-access-r8kww") pod "acedcbd0-81ff-494e-ab84-de490f2e31d0" (UID: "acedcbd0-81ff-494e-ab84-de490f2e31d0"). InnerVolumeSpecName "kube-api-access-r8kww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:56:00.243833 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.243813    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8kww\" (UniqueName: \"kubernetes.io/projected/acedcbd0-81ff-494e-ab84-de490f2e31d0-kube-api-access-r8kww\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:56:00.243833 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.243833    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:56:00.243974 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.243843    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-oauth-serving-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:56:00.243974 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.243852    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-trusted-ca-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:56:00.243974 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.243860    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-oauth-config\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:56:00.243974 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.243868    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-serving-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:56:00.243974 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.243876    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acedcbd0-81ff-494e-ab84-de490f2e31d0-console-config\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:56:00.811878 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.811845    2561 generic.go:356] "Generic (PLEG): container finished" podID="acedcbd0-81ff-494e-ab84-de490f2e31d0" containerID="a2ce1b287a06c52724943caee326e79645777ad5f0b9b7aa2ca8dbbca0efbfee" exitCode=0
Sep 14 10:56:00.812434 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.811891    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748f74fdbf-59rdf"
Sep 14 10:56:00.812434 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.811890    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748f74fdbf-59rdf" event={"ID":"acedcbd0-81ff-494e-ab84-de490f2e31d0","Type":"ContainerDied","Data":"a2ce1b287a06c52724943caee326e79645777ad5f0b9b7aa2ca8dbbca0efbfee"}
Sep 14 10:56:00.812434 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.811960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748f74fdbf-59rdf" event={"ID":"acedcbd0-81ff-494e-ab84-de490f2e31d0","Type":"ContainerDied","Data":"6d6e767db5ff371502a436eb11ac40c73f4bab139d467a686289428c7937b5bd"}
Sep 14 10:56:00.812434 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.811992    2561 scope.go:117] "RemoveContainer" containerID="a2ce1b287a06c52724943caee326e79645777ad5f0b9b7aa2ca8dbbca0efbfee"
Sep 14 10:56:00.821120 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.821105    2561 scope.go:117] "RemoveContainer" containerID="a2ce1b287a06c52724943caee326e79645777ad5f0b9b7aa2ca8dbbca0efbfee"
Sep 14 10:56:00.821340 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:56:00.821325    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ce1b287a06c52724943caee326e79645777ad5f0b9b7aa2ca8dbbca0efbfee\": container with ID starting with a2ce1b287a06c52724943caee326e79645777ad5f0b9b7aa2ca8dbbca0efbfee not found: ID does not exist" containerID="a2ce1b287a06c52724943caee326e79645777ad5f0b9b7aa2ca8dbbca0efbfee"
Sep 14 10:56:00.821389 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.821347    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ce1b287a06c52724943caee326e79645777ad5f0b9b7aa2ca8dbbca0efbfee"} err="failed to get container status \"a2ce1b287a06c52724943caee326e79645777ad5f0b9b7aa2ca8dbbca0efbfee\": rpc error: code = NotFound desc = could not find container \"a2ce1b287a06c52724943caee326e79645777ad5f0b9b7aa2ca8dbbca0efbfee\": container with ID starting with a2ce1b287a06c52724943caee326e79645777ad5f0b9b7aa2ca8dbbca0efbfee not found: ID does not exist"
Sep 14 10:56:00.855067 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.855044    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-748f74fdbf-59rdf"]
Sep 14 10:56:00.862961 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:00.862943    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-748f74fdbf-59rdf"]
Sep 14 10:56:01.802282 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:01.802250    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acedcbd0-81ff-494e-ab84-de490f2e31d0" path="/var/lib/kubelet/pods/acedcbd0-81ff-494e-ab84-de490f2e31d0/volumes"
Sep 14 10:56:06.269982 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.269951    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"]
Sep 14 10:56:06.270334 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.270260    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acedcbd0-81ff-494e-ab84-de490f2e31d0" containerName="console"
Sep 14 10:56:06.270334 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.270270    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="acedcbd0-81ff-494e-ab84-de490f2e31d0" containerName="console"
Sep 14 10:56:06.270334 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.270286    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e79e85-d179-4046-ab06-917443763e29" containerName="util"
Sep 14 10:56:06.270334 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.270292    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e79e85-d179-4046-ab06-917443763e29" containerName="util"
Sep 14 10:56:06.270334 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.270300    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e79e85-d179-4046-ab06-917443763e29" containerName="extract"
Sep 14 10:56:06.270334 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.270305    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e79e85-d179-4046-ab06-917443763e29" containerName="extract"
Sep 14 10:56:06.270334 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.270313    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e79e85-d179-4046-ab06-917443763e29" containerName="pull"
Sep 14 10:56:06.270334 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.270318    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e79e85-d179-4046-ab06-917443763e29" containerName="pull"
Sep 14 10:56:06.270614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.270377    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="acedcbd0-81ff-494e-ab84-de490f2e31d0" containerName="console"
Sep 14 10:56:06.270614 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.270387    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e79e85-d179-4046-ab06-917443763e29" containerName="extract"
Sep 14 10:56:06.274699 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.274681    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.279141 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.279120    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jqdrm\""
Sep 14 10:56:06.279270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.279180    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 14 10:56:06.279881 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.279867    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 14 10:56:06.280169 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.280153    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 14 10:56:06.291921 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.291561    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"]
Sep 14 10:56:06.393284 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.393249    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.393284 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.393298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.393524 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.393316    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41ea28d-b80f-4039-9687-8e171a0cafee-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.393524 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.393343    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.393524 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.393360    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v62dn\" (UniqueName: \"kubernetes.io/projected/b41ea28d-b80f-4039-9687-8e171a0cafee-kube-api-access-v62dn\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.393524 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.393381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41ea28d-b80f-4039-9687-8e171a0cafee-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.393524 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.393479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41ea28d-b80f-4039-9687-8e171a0cafee-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.393524 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.393509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.393779 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.393537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.494465 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.494432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41ea28d-b80f-4039-9687-8e171a0cafee-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.494465 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.494466    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.494748 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.494488    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.494748 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.494531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.494748 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.494570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.494748 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.494593    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41ea28d-b80f-4039-9687-8e171a0cafee-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.494748 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.494621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.494748 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.494644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v62dn\" (UniqueName: \"kubernetes.io/projected/b41ea28d-b80f-4039-9687-8e171a0cafee-kube-api-access-v62dn\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.494748 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.494674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41ea28d-b80f-4039-9687-8e171a0cafee-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.495070 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.495001    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.495070 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.495032    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.495178 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.495071    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.495235 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.495209    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.495464 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.495442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41ea28d-b80f-4039-9687-8e171a0cafee-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.496968 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.496942    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41ea28d-b80f-4039-9687-8e171a0cafee-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.497076 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.497059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41ea28d-b80f-4039-9687-8e171a0cafee-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.509455 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.509433    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41ea28d-b80f-4039-9687-8e171a0cafee-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.509564 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.509495    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v62dn\" (UniqueName: \"kubernetes.io/projected/b41ea28d-b80f-4039-9687-8e171a0cafee-kube-api-access-v62dn\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr\" (UID: \"b41ea28d-b80f-4039-9687-8e171a0cafee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.587339 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.587271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:06.867520 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:06.867489    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"]
Sep 14 10:56:06.868397 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:56:06.868376    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41ea28d_b80f_4039_9687_8e171a0cafee.slice/crio-aca62e5964913fb355c82dc07553d569d9ec2eed48a462ef768a750310df373f WatchSource:0}: Error finding container aca62e5964913fb355c82dc07553d569d9ec2eed48a462ef768a750310df373f: Status 404 returned error can't find the container with id aca62e5964913fb355c82dc07553d569d9ec2eed48a462ef768a750310df373f
Sep 14 10:56:07.846804 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:07.846753    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr" event={"ID":"b41ea28d-b80f-4039-9687-8e171a0cafee","Type":"ContainerStarted","Data":"aca62e5964913fb355c82dc07553d569d9ec2eed48a462ef768a750310df373f"}
Sep 14 10:56:09.010282 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.010243    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hds9s"]
Sep 14 10:56:09.015335 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.015315    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hds9s"
Sep 14 10:56:09.020567 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.020546    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-nkggv\""
Sep 14 10:56:09.021193 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.021175    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 14 10:56:09.037270 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.037245    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hds9s"]
Sep 14 10:56:09.119342 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.119311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9hs\" (UniqueName: \"kubernetes.io/projected/1c882741-8bd8-4818-bf4e-be21b6ba6599-kube-api-access-jr9hs\") pod \"odh-model-controller-7998c67dc4-hds9s\" (UID: \"1c882741-8bd8-4818-bf4e-be21b6ba6599\") " pod="opendatahub/odh-model-controller-7998c67dc4-hds9s"
Sep 14 10:56:09.119518 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.119439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c882741-8bd8-4818-bf4e-be21b6ba6599-cert\") pod \"odh-model-controller-7998c67dc4-hds9s\" (UID: \"1c882741-8bd8-4818-bf4e-be21b6ba6599\") " pod="opendatahub/odh-model-controller-7998c67dc4-hds9s"
Sep 14 10:56:09.220246 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.220210    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9hs\" (UniqueName: \"kubernetes.io/projected/1c882741-8bd8-4818-bf4e-be21b6ba6599-kube-api-access-jr9hs\") pod \"odh-model-controller-7998c67dc4-hds9s\" (UID: \"1c882741-8bd8-4818-bf4e-be21b6ba6599\") " pod="opendatahub/odh-model-controller-7998c67dc4-hds9s"
Sep 14 10:56:09.220423 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.220313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c882741-8bd8-4818-bf4e-be21b6ba6599-cert\") pod \"odh-model-controller-7998c67dc4-hds9s\" (UID: \"1c882741-8bd8-4818-bf4e-be21b6ba6599\") " pod="opendatahub/odh-model-controller-7998c67dc4-hds9s"
Sep 14 10:56:09.220495 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:56:09.220428    2561 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 14 10:56:09.220495 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:56:09.220490    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c882741-8bd8-4818-bf4e-be21b6ba6599-cert podName:1c882741-8bd8-4818-bf4e-be21b6ba6599 nodeName:}" failed. No retries permitted until 2026-09-14 10:56:09.72046861 +0000 UTC m=+474.405152978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1c882741-8bd8-4818-bf4e-be21b6ba6599-cert") pod "odh-model-controller-7998c67dc4-hds9s" (UID: "1c882741-8bd8-4818-bf4e-be21b6ba6599") : secret "odh-model-controller-webhook-cert" not found
Sep 14 10:56:09.234403 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.234369    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9hs\" (UniqueName: \"kubernetes.io/projected/1c882741-8bd8-4818-bf4e-be21b6ba6599-kube-api-access-jr9hs\") pod \"odh-model-controller-7998c67dc4-hds9s\" (UID: \"1c882741-8bd8-4818-bf4e-be21b6ba6599\") " pod="opendatahub/odh-model-controller-7998c67dc4-hds9s"
Sep 14 10:56:09.432407 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.432370    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:56:09.432514 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.432464    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:56:09.432514 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.432496    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:56:09.684078 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.683997    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5l2v9"]
Sep 14 10:56:09.687343 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.687319    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5l2v9"
Sep 14 10:56:09.694303 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.694285    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 14 10:56:09.707535 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.707514    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5l2v9"]
Sep 14 10:56:09.724035 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.724007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7827fb-8d35-490e-be09-cc313f86394d-tls-certs\") pod \"model-serving-api-54d89bd5fc-5l2v9\" (UID: \"ab7827fb-8d35-490e-be09-cc313f86394d\") " pod="opendatahub/model-serving-api-54d89bd5fc-5l2v9"
Sep 14 10:56:09.724154 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.724087    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c882741-8bd8-4818-bf4e-be21b6ba6599-cert\") pod \"odh-model-controller-7998c67dc4-hds9s\" (UID: \"1c882741-8bd8-4818-bf4e-be21b6ba6599\") " pod="opendatahub/odh-model-controller-7998c67dc4-hds9s"
Sep 14 10:56:09.724154 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.724107    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8j4\" (UniqueName: \"kubernetes.io/projected/ab7827fb-8d35-490e-be09-cc313f86394d-kube-api-access-ws8j4\") pod \"model-serving-api-54d89bd5fc-5l2v9\" (UID: \"ab7827fb-8d35-490e-be09-cc313f86394d\") " pod="opendatahub/model-serving-api-54d89bd5fc-5l2v9"
Sep 14 10:56:09.726441 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.726419    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c882741-8bd8-4818-bf4e-be21b6ba6599-cert\") pod \"odh-model-controller-7998c67dc4-hds9s\" (UID: \"1c882741-8bd8-4818-bf4e-be21b6ba6599\") " pod="opendatahub/odh-model-controller-7998c67dc4-hds9s"
Sep 14 10:56:09.825427 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.825395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8j4\" (UniqueName: \"kubernetes.io/projected/ab7827fb-8d35-490e-be09-cc313f86394d-kube-api-access-ws8j4\") pod \"model-serving-api-54d89bd5fc-5l2v9\" (UID: \"ab7827fb-8d35-490e-be09-cc313f86394d\") " pod="opendatahub/model-serving-api-54d89bd5fc-5l2v9"
Sep 14 10:56:09.825600 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.825461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7827fb-8d35-490e-be09-cc313f86394d-tls-certs\") pod \"model-serving-api-54d89bd5fc-5l2v9\" (UID: \"ab7827fb-8d35-490e-be09-cc313f86394d\") " pod="opendatahub/model-serving-api-54d89bd5fc-5l2v9"
Sep 14 10:56:09.827740 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.827701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7827fb-8d35-490e-be09-cc313f86394d-tls-certs\") pod \"model-serving-api-54d89bd5fc-5l2v9\" (UID: \"ab7827fb-8d35-490e-be09-cc313f86394d\") " pod="opendatahub/model-serving-api-54d89bd5fc-5l2v9"
Sep 14 10:56:09.834663 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.834639    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8j4\" (UniqueName: \"kubernetes.io/projected/ab7827fb-8d35-490e-be09-cc313f86394d-kube-api-access-ws8j4\") pod \"model-serving-api-54d89bd5fc-5l2v9\" (UID: \"ab7827fb-8d35-490e-be09-cc313f86394d\") " pod="opendatahub/model-serving-api-54d89bd5fc-5l2v9"
Sep 14 10:56:09.861588 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.861562    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr" event={"ID":"b41ea28d-b80f-4039-9687-8e171a0cafee","Type":"ContainerStarted","Data":"4aedaf9da46ef6e0036c5f1c4b92dce1192d57e46c5c16f485cbf9e43d80423e"}
Sep 14 10:56:09.905518 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.905457    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr" podStartSLOduration=1.343529121 podStartE2EDuration="3.905439274s" totalImagesPullingTime="2.561910153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:56:09.904734316 +0000 UTC m=+474.589418707" watchObservedRunningTime="2026-09-14 10:56:09.905439274 +0000 UTC m=+474.590123682"
Sep 14 10:56:09.926896 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.926868    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hds9s"
Sep 14 10:56:09.997814 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:09.997331    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5l2v9"
Sep 14 10:56:10.231363 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:10.231334    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hds9s"]
Sep 14 10:56:10.233260 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:56:10.233234    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c882741_8bd8_4818_bf4e_be21b6ba6599.slice/crio-7049d8b51862b351f699191d5e7139ae725fd9abcc71784d439bdd6660b3224b WatchSource:0}: Error finding container 7049d8b51862b351f699191d5e7139ae725fd9abcc71784d439bdd6660b3224b: Status 404 returned error can't find the container with id 7049d8b51862b351f699191d5e7139ae725fd9abcc71784d439bdd6660b3224b
Sep 14 10:56:10.326845 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:10.326823    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5l2v9"]
Sep 14 10:56:10.328962 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:56:10.328935    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7827fb_8d35_490e_be09_cc313f86394d.slice/crio-17cab0034e4a35809a2113690a8a5cc7c82ee5fdb608f4bcf9e071b1cf4deff5 WatchSource:0}: Error finding container 17cab0034e4a35809a2113690a8a5cc7c82ee5fdb608f4bcf9e071b1cf4deff5: Status 404 returned error can't find the container with id 17cab0034e4a35809a2113690a8a5cc7c82ee5fdb608f4bcf9e071b1cf4deff5
Sep 14 10:56:10.587933 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:10.587858    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:10.592522 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:10.592499    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:10.867835 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:10.867804    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hds9s" event={"ID":"1c882741-8bd8-4818-bf4e-be21b6ba6599","Type":"ContainerStarted","Data":"7049d8b51862b351f699191d5e7139ae725fd9abcc71784d439bdd6660b3224b"}
Sep 14 10:56:10.868929 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:10.868905    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5l2v9" event={"ID":"ab7827fb-8d35-490e-be09-cc313f86394d","Type":"ContainerStarted","Data":"17cab0034e4a35809a2113690a8a5cc7c82ee5fdb608f4bcf9e071b1cf4deff5"}
Sep 14 10:56:10.869207 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:10.869187    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:10.870112 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:10.870094    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr"
Sep 14 10:56:14.894389 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:14.894350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hds9s" event={"ID":"1c882741-8bd8-4818-bf4e-be21b6ba6599","Type":"ContainerStarted","Data":"e37fed4a33daaa8ab6b914dbf13f71b136158b174a19bded839c004c22b7e019"}
Sep 14 10:56:14.894836 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:14.894445    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-hds9s"
Sep 14 10:56:14.896015 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:14.895993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5l2v9" event={"ID":"ab7827fb-8d35-490e-be09-cc313f86394d","Type":"ContainerStarted","Data":"c6fd33ebca6695c6afaea0e160b5836849451ca6090845acd959721cd793b384"}
Sep 14 10:56:14.896120 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:14.896048    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-5l2v9"
Sep 14 10:56:14.931695 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:14.931647    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-hds9s" podStartSLOduration=3.015437054 podStartE2EDuration="6.931632989s" totalImagesPullingTime="3.916195935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:56:14.930607173 +0000 UTC m=+479.615291563" watchObservedRunningTime="2026-09-14 10:56:14.931632989 +0000 UTC m=+479.616317377"
Sep 14 10:56:14.968223 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:14.968163    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-5l2v9" podStartSLOduration=2.195020386 podStartE2EDuration="5.968145106s" totalImagesPullingTime="3.77312472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:56:14.966252543 +0000 UTC m=+479.650936934" watchObservedRunningTime="2026-09-14 10:56:14.968145106 +0000 UTC m=+479.652829495"
Sep 14 10:56:25.903365 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:25.903338    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-hds9s"
Sep 14 10:56:25.906255 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:56:25.906231    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-5l2v9"
Sep 14 10:57:17.000448 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.000369    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"]
Sep 14 10:57:17.003974 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.003959    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"
Sep 14 10:57:17.010361 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.010331    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 10:57:17.010492 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.010337    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 10:57:17.011165 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.011146    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8p4wd\""
Sep 14 10:57:17.018750 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.018706    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"]
Sep 14 10:57:17.093476 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.093443    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk\" (UID: \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"
Sep 14 10:57:17.093644 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.093492    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6bz8\" (UniqueName: \"kubernetes.io/projected/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-kube-api-access-s6bz8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk\" (UID: \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"
Sep 14 10:57:17.093644 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.093520    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk\" (UID: \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"
Sep 14 10:57:17.173399 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.173368    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"]
Sep 14 10:57:17.176927 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.176911    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"
Sep 14 10:57:17.194447 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.194426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk\" (UID: \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"
Sep 14 10:57:17.194576 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.194465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6bz8\" (UniqueName: \"kubernetes.io/projected/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-kube-api-access-s6bz8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk\" (UID: \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"
Sep 14 10:57:17.194639 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.194627    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk\" (UID: \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"
Sep 14 10:57:17.194848 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.194827    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk\" (UID: \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"
Sep 14 10:57:17.194949 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.194934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk\" (UID: \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"
Sep 14 10:57:17.206135 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.206112    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"]
Sep 14 10:57:17.218594 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.218571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6bz8\" (UniqueName: \"kubernetes.io/projected/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-kube-api-access-s6bz8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk\" (UID: \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"
Sep 14 10:57:17.295429 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.295345    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfnlt\" (UniqueName: \"kubernetes.io/projected/acce4647-6cc6-4bf3-b24c-937b6646fe44-kube-api-access-tfnlt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p\" (UID: \"acce4647-6cc6-4bf3-b24c-937b6646fe44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"
Sep 14 10:57:17.295429 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.295396    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acce4647-6cc6-4bf3-b24c-937b6646fe44-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p\" (UID: \"acce4647-6cc6-4bf3-b24c-937b6646fe44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"
Sep 14 10:57:17.295598 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.295479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acce4647-6cc6-4bf3-b24c-937b6646fe44-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p\" (UID: \"acce4647-6cc6-4bf3-b24c-937b6646fe44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"
Sep 14 10:57:17.319706 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.319685    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"
Sep 14 10:57:17.395966 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.395932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acce4647-6cc6-4bf3-b24c-937b6646fe44-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p\" (UID: \"acce4647-6cc6-4bf3-b24c-937b6646fe44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"
Sep 14 10:57:17.396101 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.396036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfnlt\" (UniqueName: \"kubernetes.io/projected/acce4647-6cc6-4bf3-b24c-937b6646fe44-kube-api-access-tfnlt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p\" (UID: \"acce4647-6cc6-4bf3-b24c-937b6646fe44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"
Sep 14 10:57:17.396101 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.396066    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acce4647-6cc6-4bf3-b24c-937b6646fe44-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p\" (UID: \"acce4647-6cc6-4bf3-b24c-937b6646fe44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"
Sep 14 10:57:17.396390 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.396366    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acce4647-6cc6-4bf3-b24c-937b6646fe44-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p\" (UID: \"acce4647-6cc6-4bf3-b24c-937b6646fe44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"
Sep 14 10:57:17.396475 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.396397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acce4647-6cc6-4bf3-b24c-937b6646fe44-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p\" (UID: \"acce4647-6cc6-4bf3-b24c-937b6646fe44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"
Sep 14 10:57:17.434950 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.434921    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"]
Sep 14 10:57:17.435372 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.435352    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfnlt\" (UniqueName: \"kubernetes.io/projected/acce4647-6cc6-4bf3-b24c-937b6646fe44-kube-api-access-tfnlt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p\" (UID: \"acce4647-6cc6-4bf3-b24c-937b6646fe44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"
Sep 14 10:57:17.440273 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.440256    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"
Sep 14 10:57:17.475950 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.475923    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"]
Sep 14 10:57:17.485648 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.485627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"
Sep 14 10:57:17.496753 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.496730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6\" (UID: \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"
Sep 14 10:57:17.496843 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.496816    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6\" (UID: \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"
Sep 14 10:57:17.496843 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.496841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fwkx\" (UniqueName: \"kubernetes.io/projected/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-kube-api-access-9fwkx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6\" (UID: \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"
Sep 14 10:57:17.594632 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.594560    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"]
Sep 14 10:57:17.597503 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.597476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6\" (UID: \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"
Sep 14 10:57:17.597646 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.597510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fwkx\" (UniqueName: \"kubernetes.io/projected/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-kube-api-access-9fwkx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6\" (UID: \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"
Sep 14 10:57:17.597646 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.597541    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6\" (UID: \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"
Sep 14 10:57:17.597870 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.597851    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6\" (UID: \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"
Sep 14 10:57:17.597943 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.597911    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6\" (UID: \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"
Sep 14 10:57:17.598389 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.598371    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"
Sep 14 10:57:17.603831 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:57:17.603807    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538749e1_90ba_4e5c_86eb_1f73a2ab2c3a.slice/crio-7f831a2bc53d2ac1aae1cfa6ca2a561518bd75001c9bc107ae9a0046d006860a WatchSource:0}: Error finding container 7f831a2bc53d2ac1aae1cfa6ca2a561518bd75001c9bc107ae9a0046d006860a: Status 404 returned error can't find the container with id 7f831a2bc53d2ac1aae1cfa6ca2a561518bd75001c9bc107ae9a0046d006860a
Sep 14 10:57:17.605800 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.605779    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"]
Sep 14 10:57:17.610475 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.610451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fwkx\" (UniqueName: \"kubernetes.io/projected/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-kube-api-access-9fwkx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6\" (UID: \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"
Sep 14 10:57:17.615726 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.615688    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"]
Sep 14 10:57:17.698266 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.698238    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4f84787-80fb-4a1f-8b23-dd605f659e33-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92\" (UID: \"d4f84787-80fb-4a1f-8b23-dd605f659e33\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"
Sep 14 10:57:17.698358 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.698300    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm5h8\" (UniqueName: \"kubernetes.io/projected/d4f84787-80fb-4a1f-8b23-dd605f659e33-kube-api-access-pm5h8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92\" (UID: \"d4f84787-80fb-4a1f-8b23-dd605f659e33\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"
Sep 14 10:57:17.698420 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.698365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4f84787-80fb-4a1f-8b23-dd605f659e33-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92\" (UID: \"d4f84787-80fb-4a1f-8b23-dd605f659e33\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"
Sep 14 10:57:17.751484 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.751456    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"
Sep 14 10:57:17.765245 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.765225    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"]
Sep 14 10:57:17.767352 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:57:17.767329    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacce4647_6cc6_4bf3_b24c_937b6646fe44.slice/crio-6cabd87f8e0cd3d452da4a450b441238495d23e5eb69c4c02818e7b72521a9d9 WatchSource:0}: Error finding container 6cabd87f8e0cd3d452da4a450b441238495d23e5eb69c4c02818e7b72521a9d9: Status 404 returned error can't find the container with id 6cabd87f8e0cd3d452da4a450b441238495d23e5eb69c4c02818e7b72521a9d9
Sep 14 10:57:17.799481 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.799446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4f84787-80fb-4a1f-8b23-dd605f659e33-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92\" (UID: \"d4f84787-80fb-4a1f-8b23-dd605f659e33\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"
Sep 14 10:57:17.799694 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.799611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4f84787-80fb-4a1f-8b23-dd605f659e33-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92\" (UID: \"d4f84787-80fb-4a1f-8b23-dd605f659e33\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"
Sep 14 10:57:17.800007 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.799960    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4f84787-80fb-4a1f-8b23-dd605f659e33-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92\" (UID: \"d4f84787-80fb-4a1f-8b23-dd605f659e33\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"
Sep 14 10:57:17.801528 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.800108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm5h8\" (UniqueName: \"kubernetes.io/projected/d4f84787-80fb-4a1f-8b23-dd605f659e33-kube-api-access-pm5h8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92\" (UID: \"d4f84787-80fb-4a1f-8b23-dd605f659e33\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"
Sep 14 10:57:17.801912 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.800788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4f84787-80fb-4a1f-8b23-dd605f659e33-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92\" (UID: \"d4f84787-80fb-4a1f-8b23-dd605f659e33\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"
Sep 14 10:57:17.813799 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.813437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm5h8\" (UniqueName: \"kubernetes.io/projected/d4f84787-80fb-4a1f-8b23-dd605f659e33-kube-api-access-pm5h8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92\" (UID: \"d4f84787-80fb-4a1f-8b23-dd605f659e33\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"
Sep 14 10:57:17.910845 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:17.910822    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"
Sep 14 10:57:18.027478 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:18.027455    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"]
Sep 14 10:57:18.028783 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:57:18.028758    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba81f64c_3f71_45fd_97a3_8ac51a7482d1.slice/crio-702e2c5c1fd62df37cb382c4589c0e1cdddc77e2b2d4f3ddd27e757ba58e2afd WatchSource:0}: Error finding container 702e2c5c1fd62df37cb382c4589c0e1cdddc77e2b2d4f3ddd27e757ba58e2afd: Status 404 returned error can't find the container with id 702e2c5c1fd62df37cb382c4589c0e1cdddc77e2b2d4f3ddd27e757ba58e2afd
Sep 14 10:57:18.182940 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:18.182918    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"]
Sep 14 10:57:18.184151 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:57:18.184130    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f84787_80fb_4a1f_8b23_dd605f659e33.slice/crio-3895e5bd2f3f11f1237cadcb53714ddd8c82a4857907f6cdd31a65177effa15f WatchSource:0}: Error finding container 3895e5bd2f3f11f1237cadcb53714ddd8c82a4857907f6cdd31a65177effa15f: Status 404 returned error can't find the container with id 3895e5bd2f3f11f1237cadcb53714ddd8c82a4857907f6cdd31a65177effa15f
Sep 14 10:57:18.202174 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:18.202078    2561 generic.go:356] "Generic (PLEG): container finished" podID="ba81f64c-3f71-45fd-97a3-8ac51a7482d1" containerID="423f51c2816c40398a2cdf7b8b53a5b7cb17cb5994c43db5b365c5be7eeb4b36" exitCode=0
Sep 14 10:57:18.202174 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:18.202113    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6" event={"ID":"ba81f64c-3f71-45fd-97a3-8ac51a7482d1","Type":"ContainerDied","Data":"423f51c2816c40398a2cdf7b8b53a5b7cb17cb5994c43db5b365c5be7eeb4b36"}
Sep 14 10:57:18.202174 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:18.202171    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6" event={"ID":"ba81f64c-3f71-45fd-97a3-8ac51a7482d1","Type":"ContainerStarted","Data":"702e2c5c1fd62df37cb382c4589c0e1cdddc77e2b2d4f3ddd27e757ba58e2afd"}
Sep 14 10:57:18.203940 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:18.203921    2561 generic.go:356] "Generic (PLEG): container finished" podID="538749e1-90ba-4e5c-86eb-1f73a2ab2c3a" containerID="8e3537473327feaebeeb77bf985b3f28a085875d11ba32a0675f8bb8c70ded10" exitCode=0
Sep 14 10:57:18.204097 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:18.204063    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk" event={"ID":"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a","Type":"ContainerDied","Data":"8e3537473327feaebeeb77bf985b3f28a085875d11ba32a0675f8bb8c70ded10"}
Sep 14 10:57:18.204173 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:18.204104    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk" event={"ID":"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a","Type":"ContainerStarted","Data":"7f831a2bc53d2ac1aae1cfa6ca2a561518bd75001c9bc107ae9a0046d006860a"}
Sep 14 10:57:18.205608 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:18.205577    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92" event={"ID":"d4f84787-80fb-4a1f-8b23-dd605f659e33","Type":"ContainerStarted","Data":"3895e5bd2f3f11f1237cadcb53714ddd8c82a4857907f6cdd31a65177effa15f"}
Sep 14 10:57:18.207223 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:18.207202    2561 generic.go:356] "Generic (PLEG): container finished" podID="acce4647-6cc6-4bf3-b24c-937b6646fe44" containerID="7ed71b2998ddfafa8946c8172efe59c0bb4472325fe3beca40929b791cc5ef2e" exitCode=0
Sep 14 10:57:18.207300 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:18.207258    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p" event={"ID":"acce4647-6cc6-4bf3-b24c-937b6646fe44","Type":"ContainerDied","Data":"7ed71b2998ddfafa8946c8172efe59c0bb4472325fe3beca40929b791cc5ef2e"}
Sep 14 10:57:18.207300 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:18.207293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p" event={"ID":"acce4647-6cc6-4bf3-b24c-937b6646fe44","Type":"ContainerStarted","Data":"6cabd87f8e0cd3d452da4a450b441238495d23e5eb69c4c02818e7b72521a9d9"}
Sep 14 10:57:19.213232 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:19.213202    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4f84787-80fb-4a1f-8b23-dd605f659e33" containerID="ac3debe46f813aa1df74610e4a3e4232f47dfabc2b64f316541c7bf853ed205b" exitCode=0
Sep 14 10:57:19.213550 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:19.213283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92" event={"ID":"d4f84787-80fb-4a1f-8b23-dd605f659e33","Type":"ContainerDied","Data":"ac3debe46f813aa1df74610e4a3e4232f47dfabc2b64f316541c7bf853ed205b"}
Sep 14 10:57:19.215468 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:19.215442    2561 generic.go:356] "Generic (PLEG): container finished" podID="acce4647-6cc6-4bf3-b24c-937b6646fe44" containerID="9a9d82ea4651101bacc127f44514b84708a1beb9a2372b079938710822bab121" exitCode=0
Sep 14 10:57:19.215569 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:19.215523    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p" event={"ID":"acce4647-6cc6-4bf3-b24c-937b6646fe44","Type":"ContainerDied","Data":"9a9d82ea4651101bacc127f44514b84708a1beb9a2372b079938710822bab121"}
Sep 14 10:57:20.222598 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:20.222520    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4f84787-80fb-4a1f-8b23-dd605f659e33" containerID="88f9901af87428131ec4d46bd7331d82e682c8bfbea760258712b0a1c2dec858" exitCode=0
Sep 14 10:57:20.223006 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:20.222597    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92" event={"ID":"d4f84787-80fb-4a1f-8b23-dd605f659e33","Type":"ContainerDied","Data":"88f9901af87428131ec4d46bd7331d82e682c8bfbea760258712b0a1c2dec858"}
Sep 14 10:57:20.224977 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:20.224958    2561 generic.go:356] "Generic (PLEG): container finished" podID="acce4647-6cc6-4bf3-b24c-937b6646fe44" containerID="3bbeaa659d48b3eb31a639e3d5450923e349f3662c9c884f8c1105d63ac93afb" exitCode=0
Sep 14 10:57:20.225063 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:20.225043    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p" event={"ID":"acce4647-6cc6-4bf3-b24c-937b6646fe44","Type":"ContainerDied","Data":"3bbeaa659d48b3eb31a639e3d5450923e349f3662c9c884f8c1105d63ac93afb"}
Sep 14 10:57:20.226721 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:20.226692    2561 generic.go:356] "Generic (PLEG): container finished" podID="538749e1-90ba-4e5c-86eb-1f73a2ab2c3a" containerID="025915d4ed33fb9a3ba68cdc548122edb6644c6ca499d242b66794ea1e8c0f85" exitCode=0
Sep 14 10:57:20.226795 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:20.226742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk" event={"ID":"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a","Type":"ContainerDied","Data":"025915d4ed33fb9a3ba68cdc548122edb6644c6ca499d242b66794ea1e8c0f85"}
Sep 14 10:57:21.233695 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.233666    2561 generic.go:356] "Generic (PLEG): container finished" podID="538749e1-90ba-4e5c-86eb-1f73a2ab2c3a" containerID="6a5acf7763c913dadb945ca239e68bae44dde50a002316bcb3c19d489cf836f8" exitCode=0
Sep 14 10:57:21.234133 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.233755    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk" event={"ID":"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a","Type":"ContainerDied","Data":"6a5acf7763c913dadb945ca239e68bae44dde50a002316bcb3c19d489cf836f8"}
Sep 14 10:57:21.235699 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.235678    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4f84787-80fb-4a1f-8b23-dd605f659e33" containerID="4e5f956ecb3b8b5b5adea6118a3fd0d5f129fcd555947cf25832d0fbc711e3a8" exitCode=0
Sep 14 10:57:21.235817 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.235750    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92" event={"ID":"d4f84787-80fb-4a1f-8b23-dd605f659e33","Type":"ContainerDied","Data":"4e5f956ecb3b8b5b5adea6118a3fd0d5f129fcd555947cf25832d0fbc711e3a8"}
Sep 14 10:57:21.364388 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.364365    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"
Sep 14 10:57:21.434302 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.434271    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfnlt\" (UniqueName: \"kubernetes.io/projected/acce4647-6cc6-4bf3-b24c-937b6646fe44-kube-api-access-tfnlt\") pod \"acce4647-6cc6-4bf3-b24c-937b6646fe44\" (UID: \"acce4647-6cc6-4bf3-b24c-937b6646fe44\") "
Sep 14 10:57:21.434430 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.434315    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acce4647-6cc6-4bf3-b24c-937b6646fe44-bundle\") pod \"acce4647-6cc6-4bf3-b24c-937b6646fe44\" (UID: \"acce4647-6cc6-4bf3-b24c-937b6646fe44\") "
Sep 14 10:57:21.434430 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.434351    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acce4647-6cc6-4bf3-b24c-937b6646fe44-util\") pod \"acce4647-6cc6-4bf3-b24c-937b6646fe44\" (UID: \"acce4647-6cc6-4bf3-b24c-937b6646fe44\") "
Sep 14 10:57:21.434854 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.434830    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acce4647-6cc6-4bf3-b24c-937b6646fe44-bundle" (OuterVolumeSpecName: "bundle") pod "acce4647-6cc6-4bf3-b24c-937b6646fe44" (UID: "acce4647-6cc6-4bf3-b24c-937b6646fe44"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:21.436331 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.436311    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acce4647-6cc6-4bf3-b24c-937b6646fe44-kube-api-access-tfnlt" (OuterVolumeSpecName: "kube-api-access-tfnlt") pod "acce4647-6cc6-4bf3-b24c-937b6646fe44" (UID: "acce4647-6cc6-4bf3-b24c-937b6646fe44"). InnerVolumeSpecName "kube-api-access-tfnlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:57:21.442120 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.442094    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acce4647-6cc6-4bf3-b24c-937b6646fe44-util" (OuterVolumeSpecName: "util") pod "acce4647-6cc6-4bf3-b24c-937b6646fe44" (UID: "acce4647-6cc6-4bf3-b24c-937b6646fe44"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:21.535932 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.535867    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfnlt\" (UniqueName: \"kubernetes.io/projected/acce4647-6cc6-4bf3-b24c-937b6646fe44-kube-api-access-tfnlt\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:57:21.535932 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.535891    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acce4647-6cc6-4bf3-b24c-937b6646fe44-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:57:21.535932 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:21.535900    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acce4647-6cc6-4bf3-b24c-937b6646fe44-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:57:22.245900 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.245869    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p"
Sep 14 10:57:22.245900 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.245892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvmw4p" event={"ID":"acce4647-6cc6-4bf3-b24c-937b6646fe44","Type":"ContainerDied","Data":"6cabd87f8e0cd3d452da4a450b441238495d23e5eb69c4c02818e7b72521a9d9"}
Sep 14 10:57:22.246347 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.245961    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cabd87f8e0cd3d452da4a450b441238495d23e5eb69c4c02818e7b72521a9d9"
Sep 14 10:57:22.385767 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.385746    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"
Sep 14 10:57:22.392339 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.392317    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"
Sep 14 10:57:22.543207 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.543136    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4f84787-80fb-4a1f-8b23-dd605f659e33-bundle\") pod \"d4f84787-80fb-4a1f-8b23-dd605f659e33\" (UID: \"d4f84787-80fb-4a1f-8b23-dd605f659e33\") "
Sep 14 10:57:22.543207 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.543169    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6bz8\" (UniqueName: \"kubernetes.io/projected/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-kube-api-access-s6bz8\") pod \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\" (UID: \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\") "
Sep 14 10:57:22.543207 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.543201    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4f84787-80fb-4a1f-8b23-dd605f659e33-util\") pod \"d4f84787-80fb-4a1f-8b23-dd605f659e33\" (UID: \"d4f84787-80fb-4a1f-8b23-dd605f659e33\") "
Sep 14 10:57:22.543454 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.543222    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-util\") pod \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\" (UID: \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\") "
Sep 14 10:57:22.543454 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.543395    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-bundle\") pod \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\" (UID: \"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a\") "
Sep 14 10:57:22.543565 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.543464    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm5h8\" (UniqueName: \"kubernetes.io/projected/d4f84787-80fb-4a1f-8b23-dd605f659e33-kube-api-access-pm5h8\") pod \"d4f84787-80fb-4a1f-8b23-dd605f659e33\" (UID: \"d4f84787-80fb-4a1f-8b23-dd605f659e33\") "
Sep 14 10:57:22.543836 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.543807    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-bundle" (OuterVolumeSpecName: "bundle") pod "538749e1-90ba-4e5c-86eb-1f73a2ab2c3a" (UID: "538749e1-90ba-4e5c-86eb-1f73a2ab2c3a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:22.543975 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.543953    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f84787-80fb-4a1f-8b23-dd605f659e33-bundle" (OuterVolumeSpecName: "bundle") pod "d4f84787-80fb-4a1f-8b23-dd605f659e33" (UID: "d4f84787-80fb-4a1f-8b23-dd605f659e33"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:22.545845 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.545826    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-kube-api-access-s6bz8" (OuterVolumeSpecName: "kube-api-access-s6bz8") pod "538749e1-90ba-4e5c-86eb-1f73a2ab2c3a" (UID: "538749e1-90ba-4e5c-86eb-1f73a2ab2c3a"). InnerVolumeSpecName "kube-api-access-s6bz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:57:22.545990 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.545963    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f84787-80fb-4a1f-8b23-dd605f659e33-kube-api-access-pm5h8" (OuterVolumeSpecName: "kube-api-access-pm5h8") pod "d4f84787-80fb-4a1f-8b23-dd605f659e33" (UID: "d4f84787-80fb-4a1f-8b23-dd605f659e33"). InnerVolumeSpecName "kube-api-access-pm5h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:57:22.552258 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.552234    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f84787-80fb-4a1f-8b23-dd605f659e33-util" (OuterVolumeSpecName: "util") pod "d4f84787-80fb-4a1f-8b23-dd605f659e33" (UID: "d4f84787-80fb-4a1f-8b23-dd605f659e33"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:22.552351 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.552316    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-util" (OuterVolumeSpecName: "util") pod "538749e1-90ba-4e5c-86eb-1f73a2ab2c3a" (UID: "538749e1-90ba-4e5c-86eb-1f73a2ab2c3a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:22.644456 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.644427    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4f84787-80fb-4a1f-8b23-dd605f659e33-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:57:22.644456 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.644453    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6bz8\" (UniqueName: \"kubernetes.io/projected/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-kube-api-access-s6bz8\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:57:22.644456 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.644464    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4f84787-80fb-4a1f-8b23-dd605f659e33-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:57:22.644659 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.644473    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:57:22.644659 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.644481    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/538749e1-90ba-4e5c-86eb-1f73a2ab2c3a-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:57:22.644659 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:22.644488    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm5h8\" (UniqueName: \"kubernetes.io/projected/d4f84787-80fb-4a1f-8b23-dd605f659e33-kube-api-access-pm5h8\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:57:23.253234 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:23.253204    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk"
Sep 14 10:57:23.253234 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:23.253217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkwkk" event={"ID":"538749e1-90ba-4e5c-86eb-1f73a2ab2c3a","Type":"ContainerDied","Data":"7f831a2bc53d2ac1aae1cfa6ca2a561518bd75001c9bc107ae9a0046d006860a"}
Sep 14 10:57:23.253725 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:23.253263    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f831a2bc53d2ac1aae1cfa6ca2a561518bd75001c9bc107ae9a0046d006860a"
Sep 14 10:57:23.255207 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:23.255189    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92"
Sep 14 10:57:23.255207 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:23.255200    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw4p92" event={"ID":"d4f84787-80fb-4a1f-8b23-dd605f659e33","Type":"ContainerDied","Data":"3895e5bd2f3f11f1237cadcb53714ddd8c82a4857907f6cdd31a65177effa15f"}
Sep 14 10:57:23.255430 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:23.255230    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3895e5bd2f3f11f1237cadcb53714ddd8c82a4857907f6cdd31a65177effa15f"
Sep 14 10:57:30.291219 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:30.291186    2561 generic.go:356] "Generic (PLEG): container finished" podID="ba81f64c-3f71-45fd-97a3-8ac51a7482d1" containerID="999550fa192dacd8ea529df850acb9c640b283669163833a4b859d8a4b4e466c" exitCode=0
Sep 14 10:57:30.291611 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:30.291235    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6" event={"ID":"ba81f64c-3f71-45fd-97a3-8ac51a7482d1","Type":"ContainerDied","Data":"999550fa192dacd8ea529df850acb9c640b283669163833a4b859d8a4b4e466c"}
Sep 14 10:57:31.299029 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:31.298992    2561 generic.go:356] "Generic (PLEG): container finished" podID="ba81f64c-3f71-45fd-97a3-8ac51a7482d1" containerID="a4b660c4f92abe2f291dc23fb636158fd7b5e87f0f5a416b05c0828e420cea59" exitCode=0
Sep 14 10:57:31.299392 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:31.299051    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6" event={"ID":"ba81f64c-3f71-45fd-97a3-8ac51a7482d1","Type":"ContainerDied","Data":"a4b660c4f92abe2f291dc23fb636158fd7b5e87f0f5a416b05c0828e420cea59"}
Sep 14 10:57:32.431877 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:32.431850    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"
Sep 14 10:57:32.525788 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:32.525758    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-util\") pod \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\" (UID: \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\") "
Sep 14 10:57:32.525952 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:32.525815    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fwkx\" (UniqueName: \"kubernetes.io/projected/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-kube-api-access-9fwkx\") pod \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\" (UID: \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\") "
Sep 14 10:57:32.525952 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:32.525843    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-bundle\") pod \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\" (UID: \"ba81f64c-3f71-45fd-97a3-8ac51a7482d1\") "
Sep 14 10:57:32.526383 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:32.526358    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-bundle" (OuterVolumeSpecName: "bundle") pod "ba81f64c-3f71-45fd-97a3-8ac51a7482d1" (UID: "ba81f64c-3f71-45fd-97a3-8ac51a7482d1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:32.527985 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:32.527961    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-kube-api-access-9fwkx" (OuterVolumeSpecName: "kube-api-access-9fwkx") pod "ba81f64c-3f71-45fd-97a3-8ac51a7482d1" (UID: "ba81f64c-3f71-45fd-97a3-8ac51a7482d1"). InnerVolumeSpecName "kube-api-access-9fwkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:57:32.530367 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:32.530347    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-util" (OuterVolumeSpecName: "util") pod "ba81f64c-3f71-45fd-97a3-8ac51a7482d1" (UID: "ba81f64c-3f71-45fd-97a3-8ac51a7482d1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:32.626670 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:32.626637    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fwkx\" (UniqueName: \"kubernetes.io/projected/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-kube-api-access-9fwkx\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:57:32.626670 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:32.626667    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:57:32.626867 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:32.626680    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba81f64c-3f71-45fd-97a3-8ac51a7482d1-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:57:33.311212 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:33.311181    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6" event={"ID":"ba81f64c-3f71-45fd-97a3-8ac51a7482d1","Type":"ContainerDied","Data":"702e2c5c1fd62df37cb382c4589c0e1cdddc77e2b2d4f3ddd27e757ba58e2afd"}
Sep 14 10:57:33.311356 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:33.311224    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mb7v6"
Sep 14 10:57:33.311356 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:33.311226    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="702e2c5c1fd62df37cb382c4589c0e1cdddc77e2b2d4f3ddd27e757ba58e2afd"
Sep 14 10:57:40.062831 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.062787    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"]
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063197    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f84787-80fb-4a1f-8b23-dd605f659e33" containerName="util"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063210    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f84787-80fb-4a1f-8b23-dd605f659e33" containerName="util"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063218    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba81f64c-3f71-45fd-97a3-8ac51a7482d1" containerName="pull"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063224    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba81f64c-3f71-45fd-97a3-8ac51a7482d1" containerName="pull"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063234    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="538749e1-90ba-4e5c-86eb-1f73a2ab2c3a" containerName="util"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063239    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="538749e1-90ba-4e5c-86eb-1f73a2ab2c3a" containerName="util"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063246    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acce4647-6cc6-4bf3-b24c-937b6646fe44" containerName="util"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063252    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="acce4647-6cc6-4bf3-b24c-937b6646fe44" containerName="util"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063257    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f84787-80fb-4a1f-8b23-dd605f659e33" containerName="pull"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063262    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f84787-80fb-4a1f-8b23-dd605f659e33" containerName="pull"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063276    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="538749e1-90ba-4e5c-86eb-1f73a2ab2c3a" containerName="extract"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063281    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="538749e1-90ba-4e5c-86eb-1f73a2ab2c3a" containerName="extract"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063288    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="538749e1-90ba-4e5c-86eb-1f73a2ab2c3a" containerName="pull"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063293    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="538749e1-90ba-4e5c-86eb-1f73a2ab2c3a" containerName="pull"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063299    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba81f64c-3f71-45fd-97a3-8ac51a7482d1" containerName="extract"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063304    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba81f64c-3f71-45fd-97a3-8ac51a7482d1" containerName="extract"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063310    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acce4647-6cc6-4bf3-b24c-937b6646fe44" containerName="extract"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063315    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="acce4647-6cc6-4bf3-b24c-937b6646fe44" containerName="extract"
Sep 14 10:57:40.063312 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063322    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba81f64c-3f71-45fd-97a3-8ac51a7482d1" containerName="util"
Sep 14 10:57:40.063994 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063327    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba81f64c-3f71-45fd-97a3-8ac51a7482d1" containerName="util"
Sep 14 10:57:40.063994 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063333    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acce4647-6cc6-4bf3-b24c-937b6646fe44" containerName="pull"
Sep 14 10:57:40.063994 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063338    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="acce4647-6cc6-4bf3-b24c-937b6646fe44" containerName="pull"
Sep 14 10:57:40.063994 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063348    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f84787-80fb-4a1f-8b23-dd605f659e33" containerName="extract"
Sep 14 10:57:40.063994 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063352    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f84787-80fb-4a1f-8b23-dd605f659e33" containerName="extract"
Sep 14 10:57:40.063994 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063406    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="acce4647-6cc6-4bf3-b24c-937b6646fe44" containerName="extract"
Sep 14 10:57:40.063994 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063415    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba81f64c-3f71-45fd-97a3-8ac51a7482d1" containerName="extract"
Sep 14 10:57:40.063994 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063422    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="538749e1-90ba-4e5c-86eb-1f73a2ab2c3a" containerName="extract"
Sep 14 10:57:40.063994 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.063430    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f84787-80fb-4a1f-8b23-dd605f659e33" containerName="extract"
Sep 14 10:57:40.066641 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.066621    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:57:40.069815 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.069796    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 14 10:57:40.070080 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.070065    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 14 10:57:40.071054 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.071035    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-khfrt\""
Sep 14 10:57:40.102507 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.102482    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"]
Sep 14 10:57:40.189547 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.189516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdf8\" (UniqueName: \"kubernetes.io/projected/eac18521-5c2b-4138-b065-b629e464b7a1-kube-api-access-jmdf8\") pod \"limitador-operator-controller-manager-74495c69f8-kj7qv\" (UID: \"eac18521-5c2b-4138-b065-b629e464b7a1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:57:40.290578 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.290544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdf8\" (UniqueName: \"kubernetes.io/projected/eac18521-5c2b-4138-b065-b629e464b7a1-kube-api-access-jmdf8\") pod \"limitador-operator-controller-manager-74495c69f8-kj7qv\" (UID: \"eac18521-5c2b-4138-b065-b629e464b7a1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:57:40.318489 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.318429    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdf8\" (UniqueName: \"kubernetes.io/projected/eac18521-5c2b-4138-b065-b629e464b7a1-kube-api-access-jmdf8\") pod \"limitador-operator-controller-manager-74495c69f8-kj7qv\" (UID: \"eac18521-5c2b-4138-b065-b629e464b7a1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:57:40.377461 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.377418    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:57:40.660810 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:40.660780    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"]
Sep 14 10:57:40.664366 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:57:40.664340    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac18521_5c2b_4138_b065_b629e464b7a1.slice/crio-ec44e801e72f6c0915a7e094a25a78671d35d6a2c679a41fc15f63bac2b353b9 WatchSource:0}: Error finding container ec44e801e72f6c0915a7e094a25a78671d35d6a2c679a41fc15f63bac2b353b9: Status 404 returned error can't find the container with id ec44e801e72f6c0915a7e094a25a78671d35d6a2c679a41fc15f63bac2b353b9
Sep 14 10:57:41.353906 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:41.353872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv" event={"ID":"eac18521-5c2b-4138-b065-b629e464b7a1","Type":"ContainerStarted","Data":"ec44e801e72f6c0915a7e094a25a78671d35d6a2c679a41fc15f63bac2b353b9"}
Sep 14 10:57:44.373123 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:44.373078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv" event={"ID":"eac18521-5c2b-4138-b065-b629e464b7a1","Type":"ContainerStarted","Data":"c8c8bc706ccb314aea10c062663404cc4de9d29a72fc77465744ad4bcc90bdca"}
Sep 14 10:57:44.373481 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:44.373143    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:57:44.414346 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:44.414300    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv" podStartSLOduration=1.195944272 podStartE2EDuration="4.414284809s" totalImagesPullingTime="3.218340537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:57:44.413466997 +0000 UTC m=+569.098151386" watchObservedRunningTime="2026-09-14 10:57:44.414284809 +0000 UTC m=+569.098969198"
Sep 14 10:57:45.106445 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:45.106405    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9cv9k"]
Sep 14 10:57:45.110053 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:45.110031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9cv9k"
Sep 14 10:57:45.112967 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:45.112945    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6sgc8\""
Sep 14 10:57:45.113242 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:45.113229    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 14 10:57:45.136138 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:45.136112    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9cv9k"]
Sep 14 10:57:45.239914 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:45.239886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8br7\" (UniqueName: \"kubernetes.io/projected/7ac50e28-8d32-43b5-80d2-4ae3ff80cb4d-kube-api-access-v8br7\") pod \"dns-operator-controller-manager-7b84cd7c9c-9cv9k\" (UID: \"7ac50e28-8d32-43b5-80d2-4ae3ff80cb4d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9cv9k"
Sep 14 10:57:45.341218 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:45.341189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8br7\" (UniqueName: \"kubernetes.io/projected/7ac50e28-8d32-43b5-80d2-4ae3ff80cb4d-kube-api-access-v8br7\") pod \"dns-operator-controller-manager-7b84cd7c9c-9cv9k\" (UID: \"7ac50e28-8d32-43b5-80d2-4ae3ff80cb4d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9cv9k"
Sep 14 10:57:45.354231 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:45.354202    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8br7\" (UniqueName: \"kubernetes.io/projected/7ac50e28-8d32-43b5-80d2-4ae3ff80cb4d-kube-api-access-v8br7\") pod \"dns-operator-controller-manager-7b84cd7c9c-9cv9k\" (UID: \"7ac50e28-8d32-43b5-80d2-4ae3ff80cb4d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9cv9k"
Sep 14 10:57:45.420097 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:45.420024    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9cv9k"
Sep 14 10:57:45.708230 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:45.708202    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9cv9k"]
Sep 14 10:57:45.709042 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:57:45.709012    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac50e28_8d32_43b5_80d2_4ae3ff80cb4d.slice/crio-dd3d1b672f88ce3788133319e58fa0ec64d07a476906b78ee57d43bfc34d3c7a WatchSource:0}: Error finding container dd3d1b672f88ce3788133319e58fa0ec64d07a476906b78ee57d43bfc34d3c7a: Status 404 returned error can't find the container with id dd3d1b672f88ce3788133319e58fa0ec64d07a476906b78ee57d43bfc34d3c7a
Sep 14 10:57:46.386204 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:46.386169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9cv9k" event={"ID":"7ac50e28-8d32-43b5-80d2-4ae3ff80cb4d","Type":"ContainerStarted","Data":"dd3d1b672f88ce3788133319e58fa0ec64d07a476906b78ee57d43bfc34d3c7a"}
Sep 14 10:57:50.723583 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:50.723549    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"]
Sep 14 10:57:50.728617 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:50.728600    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:57:50.733693 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:50.733673    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bh4bq\""
Sep 14 10:57:50.763417 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:50.763392    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"]
Sep 14 10:57:50.890021 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:50.889985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37e41627-3a2c-4357-be79-3a13d32e44ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" (UID: \"37e41627-3a2c-4357-be79-3a13d32e44ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:57:50.890021 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:50.890021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jjnw\" (UniqueName: \"kubernetes.io/projected/37e41627-3a2c-4357-be79-3a13d32e44ab-kube-api-access-8jjnw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" (UID: \"37e41627-3a2c-4357-be79-3a13d32e44ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:57:50.990531 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:50.990450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37e41627-3a2c-4357-be79-3a13d32e44ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" (UID: \"37e41627-3a2c-4357-be79-3a13d32e44ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:57:50.990531 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:50.990485    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jjnw\" (UniqueName: \"kubernetes.io/projected/37e41627-3a2c-4357-be79-3a13d32e44ab-kube-api-access-8jjnw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" (UID: \"37e41627-3a2c-4357-be79-3a13d32e44ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:57:50.990922 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:50.990899    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37e41627-3a2c-4357-be79-3a13d32e44ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" (UID: \"37e41627-3a2c-4357-be79-3a13d32e44ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:57:51.015021 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:51.015000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jjnw\" (UniqueName: \"kubernetes.io/projected/37e41627-3a2c-4357-be79-3a13d32e44ab-kube-api-access-8jjnw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" (UID: \"37e41627-3a2c-4357-be79-3a13d32e44ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:57:51.038860 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:51.038836    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:57:51.354885 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:51.354857    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"]
Sep 14 10:57:51.355884 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:57:51.355854    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e41627_3a2c_4357_be79_3a13d32e44ab.slice/crio-3f38037154c37f389352c832826619aaec8a4962412379c93f5d4d24d351e900 WatchSource:0}: Error finding container 3f38037154c37f389352c832826619aaec8a4962412379c93f5d4d24d351e900: Status 404 returned error can't find the container with id 3f38037154c37f389352c832826619aaec8a4962412379c93f5d4d24d351e900
Sep 14 10:57:51.415431 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:51.415385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb" event={"ID":"37e41627-3a2c-4357-be79-3a13d32e44ab","Type":"ContainerStarted","Data":"3f38037154c37f389352c832826619aaec8a4962412379c93f5d4d24d351e900"}
Sep 14 10:57:52.424114 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:52.424070    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9cv9k" event={"ID":"7ac50e28-8d32-43b5-80d2-4ae3ff80cb4d","Type":"ContainerStarted","Data":"6e79308227a16af43dc5ec571be943219883a11c4fdae6103366020f5fcbaecb"}
Sep 14 10:57:52.424541 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:52.424151    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9cv9k"
Sep 14 10:57:52.478356 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:52.478310    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9cv9k" podStartSLOduration=1.293357591 podStartE2EDuration="7.478295534s" totalImagesPullingTime="6.184937943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:57:52.475366703 +0000 UTC m=+577.160051091" watchObservedRunningTime="2026-09-14 10:57:52.478295534 +0000 UTC m=+577.162979934"
Sep 14 10:57:55.382857 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:55.382827    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:57:55.445191 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:55.445151    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb" event={"ID":"37e41627-3a2c-4357-be79-3a13d32e44ab","Type":"ContainerStarted","Data":"3f0ffdaada71e44133bd7f1a0f44a9598a197d7a9534b5ca48a6b33023365f41"}
Sep 14 10:57:55.445603 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:55.445489    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:57:55.489416 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:57:55.489369    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb" podStartSLOduration=1.821836288 podStartE2EDuration="5.489351493s" totalImagesPullingTime="3.667515205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:57:55.487693523 +0000 UTC m=+580.172378145" watchObservedRunningTime="2026-09-14 10:57:55.489351493 +0000 UTC m=+580.174035882"
Sep 14 10:58:03.433775 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:03.433745    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9cv9k"
Sep 14 10:58:06.457566 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:06.457536    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:58:08.978375 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:08.978339    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"]
Sep 14 10:58:08.978800 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:08.978567    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb" podUID="37e41627-3a2c-4357-be79-3a13d32e44ab" containerName="manager" containerID="cri-o://3f0ffdaada71e44133bd7f1a0f44a9598a197d7a9534b5ca48a6b33023365f41" gracePeriod=2
Sep 14 10:58:08.988163 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:08.988134    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"]
Sep 14 10:58:09.010866 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.010839    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"]
Sep 14 10:58:09.011307 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.011289    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37e41627-3a2c-4357-be79-3a13d32e44ab" containerName="manager"
Sep 14 10:58:09.011307 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.011307    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="37e41627-3a2c-4357-be79-3a13d32e44ab" containerName="manager"
Sep 14 10:58:09.011589 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.011393    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="37e41627-3a2c-4357-be79-3a13d32e44ab" containerName="manager"
Sep 14 10:58:09.014856 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.014836    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"
Sep 14 10:58:09.017189 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.017161    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="37e41627-3a2c-4357-be79-3a13d32e44ab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:58:09.023017 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.022996    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"]
Sep 14 10:58:09.023248 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.023211    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv" podUID="eac18521-5c2b-4138-b065-b629e464b7a1" containerName="manager" containerID="cri-o://c8c8bc706ccb314aea10c062663404cc4de9d29a72fc77465744ad4bcc90bdca" gracePeriod=2
Sep 14 10:58:09.032889 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.032860    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"]
Sep 14 10:58:09.036254 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.036195    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"]
Sep 14 10:58:09.036530 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.036506    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdr9b\" (UniqueName: \"kubernetes.io/projected/5d93e987-c615-4e11-972b-264f5f4289a2-kube-api-access-sdr9b\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7vzlv\" (UID: \"5d93e987-c615-4e11-972b-264f5f4289a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"
Sep 14 10:58:09.036634 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.036622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d93e987-c615-4e11-972b-264f5f4289a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7vzlv\" (UID: \"5d93e987-c615-4e11-972b-264f5f4289a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"
Sep 14 10:58:09.073229 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.073204    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6kpts"]
Sep 14 10:58:09.073793 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.073766    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac18521-5c2b-4138-b065-b629e464b7a1" containerName="manager"
Sep 14 10:58:09.073793 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.073797    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac18521-5c2b-4138-b065-b629e464b7a1" containerName="manager"
Sep 14 10:58:09.073934 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.073889    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="eac18521-5c2b-4138-b065-b629e464b7a1" containerName="manager"
Sep 14 10:58:09.077246 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.077227    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6kpts"
Sep 14 10:58:09.090663 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.090638    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6kpts"]
Sep 14 10:58:09.127068 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.127035    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="37e41627-3a2c-4357-be79-3a13d32e44ab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:58:09.130463 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.130427    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kj7qv\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="eac18521-5c2b-4138-b065-b629e464b7a1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:58:09.137846 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.137816    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d93e987-c615-4e11-972b-264f5f4289a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7vzlv\" (UID: \"5d93e987-c615-4e11-972b-264f5f4289a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"
Sep 14 10:58:09.137958 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.137883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdr9b\" (UniqueName: \"kubernetes.io/projected/5d93e987-c615-4e11-972b-264f5f4289a2-kube-api-access-sdr9b\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7vzlv\" (UID: \"5d93e987-c615-4e11-972b-264f5f4289a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"
Sep 14 10:58:09.138066 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.138045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmxts\" (UniqueName: \"kubernetes.io/projected/89c873d9-7f29-4749-8896-3a2ce1e8f485-kube-api-access-nmxts\") pod \"limitador-operator-controller-manager-74495c69f8-6kpts\" (UID: \"89c873d9-7f29-4749-8896-3a2ce1e8f485\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6kpts"
Sep 14 10:58:09.138168 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.138152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d93e987-c615-4e11-972b-264f5f4289a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7vzlv\" (UID: \"5d93e987-c615-4e11-972b-264f5f4289a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"
Sep 14 10:58:09.151023 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.150994    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdr9b\" (UniqueName: \"kubernetes.io/projected/5d93e987-c615-4e11-972b-264f5f4289a2-kube-api-access-sdr9b\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7vzlv\" (UID: \"5d93e987-c615-4e11-972b-264f5f4289a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"
Sep 14 10:58:09.238616 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.238588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmxts\" (UniqueName: \"kubernetes.io/projected/89c873d9-7f29-4749-8896-3a2ce1e8f485-kube-api-access-nmxts\") pod \"limitador-operator-controller-manager-74495c69f8-6kpts\" (UID: \"89c873d9-7f29-4749-8896-3a2ce1e8f485\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6kpts"
Sep 14 10:58:09.250000 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.249977    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmxts\" (UniqueName: \"kubernetes.io/projected/89c873d9-7f29-4749-8896-3a2ce1e8f485-kube-api-access-nmxts\") pod \"limitador-operator-controller-manager-74495c69f8-6kpts\" (UID: \"89c873d9-7f29-4749-8896-3a2ce1e8f485\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6kpts"
Sep 14 10:58:09.252224 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.252203    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:58:09.254577 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.254551    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="37e41627-3a2c-4357-be79-3a13d32e44ab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:58:09.254927 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.254913    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:58:09.256453 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.256434    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kj7qv\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="eac18521-5c2b-4138-b065-b629e464b7a1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:58:09.258490 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.258469    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="37e41627-3a2c-4357-be79-3a13d32e44ab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:58:09.260294 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.260274    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kj7qv\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="eac18521-5c2b-4138-b065-b629e464b7a1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:58:09.339486 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.339465    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37e41627-3a2c-4357-be79-3a13d32e44ab-extensions-socket-volume\") pod \"37e41627-3a2c-4357-be79-3a13d32e44ab\" (UID: \"37e41627-3a2c-4357-be79-3a13d32e44ab\") "
Sep 14 10:58:09.339608 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.339493    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jjnw\" (UniqueName: \"kubernetes.io/projected/37e41627-3a2c-4357-be79-3a13d32e44ab-kube-api-access-8jjnw\") pod \"37e41627-3a2c-4357-be79-3a13d32e44ab\" (UID: \"37e41627-3a2c-4357-be79-3a13d32e44ab\") "
Sep 14 10:58:09.339608 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.339518    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmdf8\" (UniqueName: \"kubernetes.io/projected/eac18521-5c2b-4138-b065-b629e464b7a1-kube-api-access-jmdf8\") pod \"eac18521-5c2b-4138-b065-b629e464b7a1\" (UID: \"eac18521-5c2b-4138-b065-b629e464b7a1\") "
Sep 14 10:58:09.339975 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.339953    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e41627-3a2c-4357-be79-3a13d32e44ab-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "37e41627-3a2c-4357-be79-3a13d32e44ab" (UID: "37e41627-3a2c-4357-be79-3a13d32e44ab"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:58:09.341505 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.341485    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e41627-3a2c-4357-be79-3a13d32e44ab-kube-api-access-8jjnw" (OuterVolumeSpecName: "kube-api-access-8jjnw") pod "37e41627-3a2c-4357-be79-3a13d32e44ab" (UID: "37e41627-3a2c-4357-be79-3a13d32e44ab"). InnerVolumeSpecName "kube-api-access-8jjnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:58:09.341568 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.341506    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac18521-5c2b-4138-b065-b629e464b7a1-kube-api-access-jmdf8" (OuterVolumeSpecName: "kube-api-access-jmdf8") pod "eac18521-5c2b-4138-b065-b629e464b7a1" (UID: "eac18521-5c2b-4138-b065-b629e464b7a1"). InnerVolumeSpecName "kube-api-access-jmdf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:58:09.402698 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.402677    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"
Sep 14 10:58:09.408549 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.408519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6kpts"
Sep 14 10:58:09.441084 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.441010    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37e41627-3a2c-4357-be79-3a13d32e44ab-extensions-socket-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:58:09.441084 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.441043    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jjnw\" (UniqueName: \"kubernetes.io/projected/37e41627-3a2c-4357-be79-3a13d32e44ab-kube-api-access-8jjnw\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:58:09.441084 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.441059    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmdf8\" (UniqueName: \"kubernetes.io/projected/eac18521-5c2b-4138-b065-b629e464b7a1-kube-api-access-jmdf8\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:58:09.526637 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.526564    2561 generic.go:356] "Generic (PLEG): container finished" podID="37e41627-3a2c-4357-be79-3a13d32e44ab" containerID="3f0ffdaada71e44133bd7f1a0f44a9598a197d7a9534b5ca48a6b33023365f41" exitCode=0
Sep 14 10:58:09.526637 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.526608    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:58:09.526846 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.526668    2561 scope.go:117] "RemoveContainer" containerID="3f0ffdaada71e44133bd7f1a0f44a9598a197d7a9534b5ca48a6b33023365f41"
Sep 14 10:58:09.530806 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.530773    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="37e41627-3a2c-4357-be79-3a13d32e44ab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:58:09.531642 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.531336    2561 generic.go:356] "Generic (PLEG): container finished" podID="eac18521-5c2b-4138-b065-b629e464b7a1" containerID="c8c8bc706ccb314aea10c062663404cc4de9d29a72fc77465744ad4bcc90bdca" exitCode=0
Sep 14 10:58:09.531642 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.531429    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:58:09.533799 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.533769    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kj7qv\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="eac18521-5c2b-4138-b065-b629e464b7a1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:58:09.536073 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.536045    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kj7qv\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="eac18521-5c2b-4138-b065-b629e464b7a1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:58:09.537580 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.537564    2561 scope.go:117] "RemoveContainer" containerID="3f0ffdaada71e44133bd7f1a0f44a9598a197d7a9534b5ca48a6b33023365f41"
Sep 14 10:58:09.537897 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.537877    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0ffdaada71e44133bd7f1a0f44a9598a197d7a9534b5ca48a6b33023365f41\": container with ID starting with 3f0ffdaada71e44133bd7f1a0f44a9598a197d7a9534b5ca48a6b33023365f41 not found: ID does not exist" containerID="3f0ffdaada71e44133bd7f1a0f44a9598a197d7a9534b5ca48a6b33023365f41"
Sep 14 10:58:09.537985 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.537905    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0ffdaada71e44133bd7f1a0f44a9598a197d7a9534b5ca48a6b33023365f41"} err="failed to get container status \"3f0ffdaada71e44133bd7f1a0f44a9598a197d7a9534b5ca48a6b33023365f41\": rpc error: code = NotFound desc = could not find container \"3f0ffdaada71e44133bd7f1a0f44a9598a197d7a9534b5ca48a6b33023365f41\": container with ID starting with 3f0ffdaada71e44133bd7f1a0f44a9598a197d7a9534b5ca48a6b33023365f41 not found: ID does not exist"
Sep 14 10:58:09.537985 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.537921    2561 scope.go:117] "RemoveContainer" containerID="c8c8bc706ccb314aea10c062663404cc4de9d29a72fc77465744ad4bcc90bdca"
Sep 14 10:58:09.538619 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.538584    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="37e41627-3a2c-4357-be79-3a13d32e44ab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:58:09.540588 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.540564    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="37e41627-3a2c-4357-be79-3a13d32e44ab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:58:09.542490 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.542462    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kj7qv\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="eac18521-5c2b-4138-b065-b629e464b7a1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:58:09.544438 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.544418    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8cfsb\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="37e41627-3a2c-4357-be79-3a13d32e44ab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8cfsb"
Sep 14 10:58:09.545821 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.545806    2561 scope.go:117] "RemoveContainer" containerID="c8c8bc706ccb314aea10c062663404cc4de9d29a72fc77465744ad4bcc90bdca"
Sep 14 10:58:09.546084 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.546068    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c8bc706ccb314aea10c062663404cc4de9d29a72fc77465744ad4bcc90bdca\": container with ID starting with c8c8bc706ccb314aea10c062663404cc4de9d29a72fc77465744ad4bcc90bdca not found: ID does not exist" containerID="c8c8bc706ccb314aea10c062663404cc4de9d29a72fc77465744ad4bcc90bdca"
Sep 14 10:58:09.546141 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.546093    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c8bc706ccb314aea10c062663404cc4de9d29a72fc77465744ad4bcc90bdca"} err="failed to get container status \"c8c8bc706ccb314aea10c062663404cc4de9d29a72fc77465744ad4bcc90bdca\": rpc error: code = NotFound desc = could not find container \"c8c8bc706ccb314aea10c062663404cc4de9d29a72fc77465744ad4bcc90bdca\": container with ID starting with c8c8bc706ccb314aea10c062663404cc4de9d29a72fc77465744ad4bcc90bdca not found: ID does not exist"
Sep 14 10:58:09.546563 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:09.546541    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kj7qv\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="eac18521-5c2b-4138-b065-b629e464b7a1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj7qv"
Sep 14 10:58:09.691817 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.691796    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6kpts"]
Sep 14 10:58:09.693182 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:58:09.693162    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c873d9_7f29_4749_8896_3a2ce1e8f485.slice/crio-377487f77648d3a110797a20c2cf3f6a5f9b14818083ed8990f7ff8140efcefe WatchSource:0}: Error finding container 377487f77648d3a110797a20c2cf3f6a5f9b14818083ed8990f7ff8140efcefe: Status 404 returned error can't find the container with id 377487f77648d3a110797a20c2cf3f6a5f9b14818083ed8990f7ff8140efcefe
Sep 14 10:58:09.693651 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:58:09.693637    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d93e987_c615_4e11_972b_264f5f4289a2.slice/crio-efa723bef46f03492b76d402fee34062e2ca53d07851c688dc9d750a2256e13a WatchSource:0}: Error finding container efa723bef46f03492b76d402fee34062e2ca53d07851c688dc9d750a2256e13a: Status 404 returned error can't find the container with id efa723bef46f03492b76d402fee34062e2ca53d07851c688dc9d750a2256e13a
Sep 14 10:58:09.694963 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.694945    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"]
Sep 14 10:58:09.803454 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.803388    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37e41627-3a2c-4357-be79-3a13d32e44ab" path="/var/lib/kubelet/pods/37e41627-3a2c-4357-be79-3a13d32e44ab/volumes"
Sep 14 10:58:09.803693 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:09.803681    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eac18521-5c2b-4138-b065-b629e464b7a1" path="/var/lib/kubelet/pods/eac18521-5c2b-4138-b065-b629e464b7a1/volumes"
Sep 14 10:58:10.538498 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:10.538462    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6kpts" event={"ID":"89c873d9-7f29-4749-8896-3a2ce1e8f485","Type":"ContainerStarted","Data":"238163a1f679363f952a1c9969599a13b0553b30311fa7e73b042910a847ef81"}
Sep 14 10:58:10.538898 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:10.538524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6kpts" event={"ID":"89c873d9-7f29-4749-8896-3a2ce1e8f485","Type":"ContainerStarted","Data":"377487f77648d3a110797a20c2cf3f6a5f9b14818083ed8990f7ff8140efcefe"}
Sep 14 10:58:10.538898 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:10.538551    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6kpts"
Sep 14 10:58:10.542045 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:10.542018    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv" event={"ID":"5d93e987-c615-4e11-972b-264f5f4289a2","Type":"ContainerStarted","Data":"d4be7a8a44ae321235d9dd3032f549e8a77a89b9947cb66c1ff845c0265f8514"}
Sep 14 10:58:10.542204 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:10.542066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv" event={"ID":"5d93e987-c615-4e11-972b-264f5f4289a2","Type":"ContainerStarted","Data":"efa723bef46f03492b76d402fee34062e2ca53d07851c688dc9d750a2256e13a"}
Sep 14 10:58:10.542204 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:10.542098    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"
Sep 14 10:58:10.561040 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:10.560996    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6kpts" podStartSLOduration=1.560980221 podStartE2EDuration="1.560980221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:58:10.559024944 +0000 UTC m=+595.243709334" watchObservedRunningTime="2026-09-14 10:58:10.560980221 +0000 UTC m=+595.245664610"
Sep 14 10:58:10.592728 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:10.592668    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv" podStartSLOduration=2.592654096 podStartE2EDuration="2.592654096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:58:10.590883933 +0000 UTC m=+595.275568337" watchObservedRunningTime="2026-09-14 10:58:10.592654096 +0000 UTC m=+595.277338484"
Sep 14 10:58:15.787748 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:15.787708    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 10:58:15.791336 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:15.791312    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 10:58:21.549668 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:21.549634    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6kpts"
Sep 14 10:58:21.552151 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:21.549704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"
Sep 14 10:58:24.867205 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:24.867172    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"]
Sep 14 10:58:24.867966 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:24.867931    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv" podUID="5d93e987-c615-4e11-972b-264f5f4289a2" containerName="manager" containerID="cri-o://d4be7a8a44ae321235d9dd3032f549e8a77a89b9947cb66c1ff845c0265f8514" gracePeriod=10
Sep 14 10:58:25.127856 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.127826    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"]
Sep 14 10:58:25.149763 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.149739    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"
Sep 14 10:58:25.155237 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.155213    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"]
Sep 14 10:58:25.157835 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.157815    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"
Sep 14 10:58:25.272906 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.272877    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d93e987-c615-4e11-972b-264f5f4289a2-extensions-socket-volume\") pod \"5d93e987-c615-4e11-972b-264f5f4289a2\" (UID: \"5d93e987-c615-4e11-972b-264f5f4289a2\") "
Sep 14 10:58:25.273086 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.272977    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdr9b\" (UniqueName: \"kubernetes.io/projected/5d93e987-c615-4e11-972b-264f5f4289a2-kube-api-access-sdr9b\") pod \"5d93e987-c615-4e11-972b-264f5f4289a2\" (UID: \"5d93e987-c615-4e11-972b-264f5f4289a2\") "
Sep 14 10:58:25.273086 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.273083    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87dc3b98-f8dc-41f8-aa25-9fd9f802fca5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d5zx4\" (UID: \"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"
Sep 14 10:58:25.273219 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.273110    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dckfd\" (UniqueName: \"kubernetes.io/projected/87dc3b98-f8dc-41f8-aa25-9fd9f802fca5-kube-api-access-dckfd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d5zx4\" (UID: \"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"
Sep 14 10:58:25.273321 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.273295    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d93e987-c615-4e11-972b-264f5f4289a2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5d93e987-c615-4e11-972b-264f5f4289a2" (UID: "5d93e987-c615-4e11-972b-264f5f4289a2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:58:25.274967 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.274940    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d93e987-c615-4e11-972b-264f5f4289a2-kube-api-access-sdr9b" (OuterVolumeSpecName: "kube-api-access-sdr9b") pod "5d93e987-c615-4e11-972b-264f5f4289a2" (UID: "5d93e987-c615-4e11-972b-264f5f4289a2"). InnerVolumeSpecName "kube-api-access-sdr9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:58:25.374064 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.374000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87dc3b98-f8dc-41f8-aa25-9fd9f802fca5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d5zx4\" (UID: \"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"
Sep 14 10:58:25.374064 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.374037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dckfd\" (UniqueName: \"kubernetes.io/projected/87dc3b98-f8dc-41f8-aa25-9fd9f802fca5-kube-api-access-dckfd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d5zx4\" (UID: \"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"
Sep 14 10:58:25.374205 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.374093    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdr9b\" (UniqueName: \"kubernetes.io/projected/5d93e987-c615-4e11-972b-264f5f4289a2-kube-api-access-sdr9b\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:58:25.374205 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.374102    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d93e987-c615-4e11-972b-264f5f4289a2-extensions-socket-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 10:58:25.374350 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.374329    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87dc3b98-f8dc-41f8-aa25-9fd9f802fca5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d5zx4\" (UID: \"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"
Sep 14 10:58:25.384934 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.384912    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dckfd\" (UniqueName: \"kubernetes.io/projected/87dc3b98-f8dc-41f8-aa25-9fd9f802fca5-kube-api-access-dckfd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d5zx4\" (UID: \"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"
Sep 14 10:58:25.461608 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.461573    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"
Sep 14 10:58:25.627097 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.627018    2561 generic.go:356] "Generic (PLEG): container finished" podID="5d93e987-c615-4e11-972b-264f5f4289a2" containerID="d4be7a8a44ae321235d9dd3032f549e8a77a89b9947cb66c1ff845c0265f8514" exitCode=0
Sep 14 10:58:25.627097 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.627089    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"
Sep 14 10:58:25.627292 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.627101    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv" event={"ID":"5d93e987-c615-4e11-972b-264f5f4289a2","Type":"ContainerDied","Data":"d4be7a8a44ae321235d9dd3032f549e8a77a89b9947cb66c1ff845c0265f8514"}
Sep 14 10:58:25.627292 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.627176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv" event={"ID":"5d93e987-c615-4e11-972b-264f5f4289a2","Type":"ContainerDied","Data":"efa723bef46f03492b76d402fee34062e2ca53d07851c688dc9d750a2256e13a"}
Sep 14 10:58:25.627292 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.627199    2561 scope.go:117] "RemoveContainer" containerID="d4be7a8a44ae321235d9dd3032f549e8a77a89b9947cb66c1ff845c0265f8514"
Sep 14 10:58:25.640120 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.640101    2561 scope.go:117] "RemoveContainer" containerID="d4be7a8a44ae321235d9dd3032f549e8a77a89b9947cb66c1ff845c0265f8514"
Sep 14 10:58:25.640414 ip-10-0-136-89 kubenswrapper[2561]: E0914 10:58:25.640393    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4be7a8a44ae321235d9dd3032f549e8a77a89b9947cb66c1ff845c0265f8514\": container with ID starting with d4be7a8a44ae321235d9dd3032f549e8a77a89b9947cb66c1ff845c0265f8514 not found: ID does not exist" containerID="d4be7a8a44ae321235d9dd3032f549e8a77a89b9947cb66c1ff845c0265f8514"
Sep 14 10:58:25.640516 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.640421    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4be7a8a44ae321235d9dd3032f549e8a77a89b9947cb66c1ff845c0265f8514"} err="failed to get container status \"d4be7a8a44ae321235d9dd3032f549e8a77a89b9947cb66c1ff845c0265f8514\": rpc error: code = NotFound desc = could not find container \"d4be7a8a44ae321235d9dd3032f549e8a77a89b9947cb66c1ff845c0265f8514\": container with ID starting with d4be7a8a44ae321235d9dd3032f549e8a77a89b9947cb66c1ff845c0265f8514 not found: ID does not exist"
Sep 14 10:58:25.656971 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.656948    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"]
Sep 14 10:58:25.663203 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.663183    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7vzlv"]
Sep 14 10:58:25.758257 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.758232    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"]
Sep 14 10:58:25.759217 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:58:25.759197    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dc3b98_f8dc_41f8_aa25_9fd9f802fca5.slice/crio-1d1871026816ee8354f2032fb644b2102990b66ba9fc16a22b41b0acce4d4633 WatchSource:0}: Error finding container 1d1871026816ee8354f2032fb644b2102990b66ba9fc16a22b41b0acce4d4633: Status 404 returned error can't find the container with id 1d1871026816ee8354f2032fb644b2102990b66ba9fc16a22b41b0acce4d4633
Sep 14 10:58:25.802180 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:25.802158    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d93e987-c615-4e11-972b-264f5f4289a2" path="/var/lib/kubelet/pods/5d93e987-c615-4e11-972b-264f5f4289a2/volumes"
Sep 14 10:58:26.635340 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:26.635294    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4" event={"ID":"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5","Type":"ContainerStarted","Data":"bf0e6ea807af7f7048f9a52dba962d0d7f1a325ae93da0290fce740b7808502e"}
Sep 14 10:58:26.635799 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:26.635350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4" event={"ID":"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5","Type":"ContainerStarted","Data":"1d1871026816ee8354f2032fb644b2102990b66ba9fc16a22b41b0acce4d4633"}
Sep 14 10:58:26.635799 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:26.635370    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"
Sep 14 10:58:26.728627 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:26.728578    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4" podStartSLOduration=1.728557195 podStartE2EDuration="1.728557195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:58:26.726098204 +0000 UTC m=+611.410782593" watchObservedRunningTime="2026-09-14 10:58:26.728557195 +0000 UTC m=+611.413241581"
Sep 14 10:58:37.642857 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:37.642823    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"
Sep 14 10:58:57.986889 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:57.986855    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 10:58:57.987253 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:57.987223    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d93e987-c615-4e11-972b-264f5f4289a2" containerName="manager"
Sep 14 10:58:57.987253 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:57.987233    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d93e987-c615-4e11-972b-264f5f4289a2" containerName="manager"
Sep 14 10:58:57.987326 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:57.987308    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d93e987-c615-4e11-972b-264f5f4289a2" containerName="manager"
Sep 14 10:58:57.990526 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:57.990507    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-w6npf"
Sep 14 10:58:57.993834 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:57.993815    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 14 10:58:57.993939 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:57.993879    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-j858j\""
Sep 14 10:58:58.000178 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:58.000157    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 10:58:58.032271 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:58.032242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1dd58c50-d680-4fe6-92cc-b2c8a3adbca3-config-file\") pod \"limitador-limitador-84994f7ffc-w6npf\" (UID: \"1dd58c50-d680-4fe6-92cc-b2c8a3adbca3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w6npf"
Sep 14 10:58:58.032400 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:58.032286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zc45\" (UniqueName: \"kubernetes.io/projected/1dd58c50-d680-4fe6-92cc-b2c8a3adbca3-kube-api-access-7zc45\") pod \"limitador-limitador-84994f7ffc-w6npf\" (UID: \"1dd58c50-d680-4fe6-92cc-b2c8a3adbca3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w6npf"
Sep 14 10:58:58.036690 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:58.036666    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 10:58:58.133051 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:58.133014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1dd58c50-d680-4fe6-92cc-b2c8a3adbca3-config-file\") pod \"limitador-limitador-84994f7ffc-w6npf\" (UID: \"1dd58c50-d680-4fe6-92cc-b2c8a3adbca3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w6npf"
Sep 14 10:58:58.133206 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:58.133075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zc45\" (UniqueName: \"kubernetes.io/projected/1dd58c50-d680-4fe6-92cc-b2c8a3adbca3-kube-api-access-7zc45\") pod \"limitador-limitador-84994f7ffc-w6npf\" (UID: \"1dd58c50-d680-4fe6-92cc-b2c8a3adbca3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w6npf"
Sep 14 10:58:58.133582 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:58.133562    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1dd58c50-d680-4fe6-92cc-b2c8a3adbca3-config-file\") pod \"limitador-limitador-84994f7ffc-w6npf\" (UID: \"1dd58c50-d680-4fe6-92cc-b2c8a3adbca3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w6npf"
Sep 14 10:58:58.144292 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:58.144268    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zc45\" (UniqueName: \"kubernetes.io/projected/1dd58c50-d680-4fe6-92cc-b2c8a3adbca3-kube-api-access-7zc45\") pod \"limitador-limitador-84994f7ffc-w6npf\" (UID: \"1dd58c50-d680-4fe6-92cc-b2c8a3adbca3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-w6npf"
Sep 14 10:58:58.302344 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:58.302269    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-w6npf"
Sep 14 10:58:58.590431 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:58.590400    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 10:58:58.591028 ip-10-0-136-89 kubenswrapper[2561]: W0914 10:58:58.591004    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd58c50_d680_4fe6_92cc_b2c8a3adbca3.slice/crio-6732127c3549583e73cd3010b184665f4287ee4b04c6ea2235f24aeaf1dbc777 WatchSource:0}: Error finding container 6732127c3549583e73cd3010b184665f4287ee4b04c6ea2235f24aeaf1dbc777: Status 404 returned error can't find the container with id 6732127c3549583e73cd3010b184665f4287ee4b04c6ea2235f24aeaf1dbc777
Sep 14 10:58:58.818214 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:58:58.818177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-w6npf" event={"ID":"1dd58c50-d680-4fe6-92cc-b2c8a3adbca3","Type":"ContainerStarted","Data":"6732127c3549583e73cd3010b184665f4287ee4b04c6ea2235f24aeaf1dbc777"}
Sep 14 10:59:00.830154 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:59:00.830065    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-w6npf" event={"ID":"1dd58c50-d680-4fe6-92cc-b2c8a3adbca3","Type":"ContainerStarted","Data":"d31ded26c59a312e795fa6fac85e112925e967b25528681c31080086063b822e"}
Sep 14 10:59:00.830154 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:59:00.830146    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-w6npf"
Sep 14 10:59:00.850471 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:59:00.850420    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-w6npf" podStartSLOduration=1.93090186 podStartE2EDuration="3.850404848s" totalImagesPullingTime="1.919502988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:59:00.847968964 +0000 UTC m=+645.532653344" watchObservedRunningTime="2026-09-14 10:59:00.850404848 +0000 UTC m=+645.535089236"
Sep 14 10:59:11.836156 ip-10-0-136-89 kubenswrapper[2561]: I0914 10:59:11.836128    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-w6npf"
Sep 14 11:00:12.179314 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:00:12.179229    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:00:28.936804 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:00:28.936761    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:00:34.507675 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:00:34.507640    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:01:09.113778 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:01:09.113744    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:01:13.823152 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:01:13.823121    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:01:21.680014 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:01:21.679977    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:01:36.307494 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:01:36.307460    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:01:45.224069 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:01:45.223979    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:03:15.827857 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:15.827789    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 11:03:15.830539 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:15.830516    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 11:03:39.116211 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.116180    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"]
Sep 14 11:03:39.119741 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.119703    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.125843 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.125821    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7984b1-openshift-default-dockercfg-krqp2\""
Sep 14 11:03:39.135118 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.135094    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"]
Sep 14 11:03:39.166159 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.166128    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-workload-socket\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.166266 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.166181    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-workload-certs\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.166266 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.166242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-data\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.166364 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.166274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-podinfo\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.166364 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.166294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-envoy\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.166364 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.166308    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istiod-ca-cert\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.166364 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.166338    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-token\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.166499 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.166382    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22sbl\" (UniqueName: \"kubernetes.io/projected/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-kube-api-access-22sbl\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.166499 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.166450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-credential-socket\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.200073 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.200049    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"]
Sep 14 11:03:39.203462 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.203447    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.206257 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.206237    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ebf73-openshift-default-dockercfg-sjn62\""
Sep 14 11:03:39.218008 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.217987    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"]
Sep 14 11:03:39.266998 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.266946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-credential-socket\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.267160 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.267031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-workload-socket\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.267160 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.267090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-workload-certs\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.267160 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.267142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-data\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.267530 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.267502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-workload-certs\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.267614 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.267553    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-credential-socket\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.267614 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.267600    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-podinfo\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.267742 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.267644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-envoy\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.267742 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.267673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istiod-ca-cert\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.267742 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.267729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-token\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.267907 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.267790    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22sbl\" (UniqueName: \"kubernetes.io/projected/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-kube-api-access-22sbl\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.268508 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.268480    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-data\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.268768 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.268747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-workload-socket\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.268862 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.268754    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istiod-ca-cert\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.272727 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.271220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-envoy\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.275795 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.274635    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-podinfo\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.279084 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.279028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22sbl\" (UniqueName: \"kubernetes.io/projected/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-kube-api-access-22sbl\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.280976 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.280926    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-token\") pod \"e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") " pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.370529 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.370442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-workload-certs\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.370529 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.370492    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-envoy\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.370529 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.370521    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-credential-socket\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.370790 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.370541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-podinfo\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.370790 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.370569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg8nn\" (UniqueName: \"kubernetes.io/projected/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-kube-api-access-vg8nn\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.370790 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.370649    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-data\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.370790 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.370682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-token\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.370790 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.370781    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istiod-ca-cert\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.370989 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.370819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-workload-socket\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.430353 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.430329    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:39.475612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.473009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-workload-certs\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.475612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.473066    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-envoy\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.475612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.473099    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-credential-socket\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.475612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.473125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-podinfo\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.475612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.473172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg8nn\" (UniqueName: \"kubernetes.io/projected/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-kube-api-access-vg8nn\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.475612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.473213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-data\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.475612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.473244    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-token\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.475612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.473308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istiod-ca-cert\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.475612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.473336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-workload-socket\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.475612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.473773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-workload-socket\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.475612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.474022    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-workload-certs\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.475612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.474828    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-credential-socket\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.475612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.475295    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-data\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.477022 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.475919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istiod-ca-cert\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.477022 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.476736    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-envoy\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.478340 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.478322    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-podinfo\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.489367 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.489343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-token\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.492385 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.492157    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg8nn\" (UniqueName: \"kubernetes.io/projected/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-kube-api-access-vg8nn\") pod \"e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") " pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.515654 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.515631    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:39.723160 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.723135    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"]
Sep 14 11:03:39.724855 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:03:39.724830    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51381b7_acf4_4af2_b15a_1dc7119d2c6c.slice/crio-4d1a653f1ff787cdbf0f479caf8f150a0f8727b4b51db6a947fe21c04bd2ea8c WatchSource:0}: Error finding container 4d1a653f1ff787cdbf0f479caf8f150a0f8727b4b51db6a947fe21c04bd2ea8c: Status 404 returned error can't find the container with id 4d1a653f1ff787cdbf0f479caf8f150a0f8727b4b51db6a947fe21c04bd2ea8c
Sep 14 11:03:39.726706 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.726675    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:03:39.726823 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.726768    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:03:39.726823 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.726814    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:03:39.800531 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:03:39.800492    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6291a6_7a72_48f0_bf8c_80dd9ed61138.slice/crio-4a123b1fa5cd90a6611aa571c8ea20aedd0ebc4b8a1642cf3b396e839a25e37f WatchSource:0}: Error finding container 4a123b1fa5cd90a6611aa571c8ea20aedd0ebc4b8a1642cf3b396e839a25e37f: Status 404 returned error can't find the container with id 4a123b1fa5cd90a6611aa571c8ea20aedd0ebc4b8a1642cf3b396e839a25e37f
Sep 14 11:03:39.803840 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.803805    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:03:39.803930 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.803885    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:03:39.803984 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.803932    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:03:39.805037 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:39.805010    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"]
Sep 14 11:03:40.351201 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.351165    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" event={"ID":"0d6291a6-7a72-48f0-bf8c-80dd9ed61138","Type":"ContainerStarted","Data":"4522dcc307674370c5d536d58e1c68c8de1d0c9b47e810d362c886096f377d12"}
Sep 14 11:03:40.351631 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.351225    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" event={"ID":"0d6291a6-7a72-48f0-bf8c-80dd9ed61138","Type":"ContainerStarted","Data":"4a123b1fa5cd90a6611aa571c8ea20aedd0ebc4b8a1642cf3b396e839a25e37f"}
Sep 14 11:03:40.352899 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.352872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" event={"ID":"f51381b7-acf4-4af2-b15a-1dc7119d2c6c","Type":"ContainerStarted","Data":"44c6aecdf266989e29021f14b68dfe1e0a8aac4c9f2d52e46af12ac452adb2d1"}
Sep 14 11:03:40.353042 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.352913    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" event={"ID":"f51381b7-acf4-4af2-b15a-1dc7119d2c6c","Type":"ContainerStarted","Data":"4d1a653f1ff787cdbf0f479caf8f150a0f8727b4b51db6a947fe21c04bd2ea8c"}
Sep 14 11:03:40.380736 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.380680    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" podStartSLOduration=1.380665736 podStartE2EDuration="1.380665736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:40.378779405 +0000 UTC m=+925.063463847" watchObservedRunningTime="2026-09-14 11:03:40.380665736 +0000 UTC m=+925.065350125"
Sep 14 11:03:40.395528 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.395503    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7854-4be324"]
Sep 14 11:03:40.398973 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.398958    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-4be324"
Sep 14 11:03:40.402265 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.402248    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-stw9c\""
Sep 14 11:03:40.409265 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.409244    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7854-4be324"]
Sep 14 11:03:40.414817 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.414776    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" podStartSLOduration=1.414762277 podStartE2EDuration="1.414762277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:40.411670829 +0000 UTC m=+925.096355245" watchObservedRunningTime="2026-09-14 11:03:40.414762277 +0000 UTC m=+925.099446680"
Sep 14 11:03:40.430608 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.430570    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:40.435303 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.435286    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:40.483472 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.483443    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2phhc\" (UniqueName: \"kubernetes.io/projected/490aff59-dc96-4120-8084-9bf277402b27-kube-api-access-2phhc\") pod \"test-curl-7854-4be324\" (UID: \"490aff59-dc96-4120-8084-9bf277402b27\") " pod="opendatahub/test-curl-7854-4be324"
Sep 14 11:03:40.517016 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.516989    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:40.521684 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.521668    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:40.584821 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.584794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2phhc\" (UniqueName: \"kubernetes.io/projected/490aff59-dc96-4120-8084-9bf277402b27-kube-api-access-2phhc\") pod \"test-curl-7854-4be324\" (UID: \"490aff59-dc96-4120-8084-9bf277402b27\") " pod="opendatahub/test-curl-7854-4be324"
Sep 14 11:03:40.594312 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.594291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2phhc\" (UniqueName: \"kubernetes.io/projected/490aff59-dc96-4120-8084-9bf277402b27-kube-api-access-2phhc\") pod \"test-curl-7854-4be324\" (UID: \"490aff59-dc96-4120-8084-9bf277402b27\") " pod="opendatahub/test-curl-7854-4be324"
Sep 14 11:03:40.709534 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.709509    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-4be324"
Sep 14 11:03:40.992161 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.992140    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7854-4be324"]
Sep 14 11:03:40.994183 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:03:40.994159    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod490aff59_dc96_4120_8084_9bf277402b27.slice/crio-7120e7cffdbe8c950b96a18b32e85cecdef04885c1f7cd9738163679512abe28 WatchSource:0}: Error finding container 7120e7cffdbe8c950b96a18b32e85cecdef04885c1f7cd9738163679512abe28: Status 404 returned error can't find the container with id 7120e7cffdbe8c950b96a18b32e85cecdef04885c1f7cd9738163679512abe28
Sep 14 11:03:40.996240 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:40.996223    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 11:03:41.361747 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:41.361702    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7854-4be324" event={"ID":"490aff59-dc96-4120-8084-9bf277402b27","Type":"ContainerStarted","Data":"7120e7cffdbe8c950b96a18b32e85cecdef04885c1f7cd9738163679512abe28"}
Sep 14 11:03:41.362193 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:41.362137    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:41.362193 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:41.362180    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:41.362895 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:41.362876    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:03:41.363017 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:41.363001    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:03:42.465905 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:42.465869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:03:43.383024 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:43.382985    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7854-4be324" event={"ID":"490aff59-dc96-4120-8084-9bf277402b27","Type":"ContainerStarted","Data":"316824e2dd6fb72a7ed92c1d6bca26491bf7e98be401b7c59f7a2fed4d0a227c"}
Sep 14 11:03:43.427621 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:43.427576    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7854-4be324" podStartSLOduration=1.671769925 podStartE2EDuration="3.427562488s" totalImagesPullingTime="1.755792563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:43.425613809 +0000 UTC m=+928.110298199" watchObservedRunningTime="2026-09-14 11:03:43.427562488 +0000 UTC m=+928.112246874"
Sep 14 11:03:44.091943 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:44.091909    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7854-4be324"]
Sep 14 11:03:44.095329 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:44.095300    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7854-4be324"]
Sep 14 11:03:44.219784 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:44.219753    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7854-02a1ee"]
Sep 14 11:03:44.220167 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:44.220153    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="490aff59-dc96-4120-8084-9bf277402b27" containerName="test-curl-7854-4be324"
Sep 14 11:03:44.220224 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:44.220169    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="490aff59-dc96-4120-8084-9bf277402b27" containerName="test-curl-7854-4be324"
Sep 14 11:03:44.220260 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:44.220233    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="490aff59-dc96-4120-8084-9bf277402b27" containerName="test-curl-7854-4be324"
Sep 14 11:03:44.223384 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:44.223365    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-02a1ee"
Sep 14 11:03:44.233069 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:44.233042    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7854-02a1ee"]
Sep 14 11:03:44.316922 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:44.316892    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65lkc\" (UniqueName: \"kubernetes.io/projected/674e1bec-4721-46dc-85ee-3f7399e8e77d-kube-api-access-65lkc\") pod \"test-curl-7854-02a1ee\" (UID: \"674e1bec-4721-46dc-85ee-3f7399e8e77d\") " pod="opendatahub/test-curl-7854-02a1ee"
Sep 14 11:03:44.388151 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:03:44.388124    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7854-4be324" container="test-curl-7854-4be324"
Sep 14 11:03:44.418003 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:44.417980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65lkc\" (UniqueName: \"kubernetes.io/projected/674e1bec-4721-46dc-85ee-3f7399e8e77d-kube-api-access-65lkc\") pod \"test-curl-7854-02a1ee\" (UID: \"674e1bec-4721-46dc-85ee-3f7399e8e77d\") " pod="opendatahub/test-curl-7854-02a1ee"
Sep 14 11:03:44.426518 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:44.426490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65lkc\" (UniqueName: \"kubernetes.io/projected/674e1bec-4721-46dc-85ee-3f7399e8e77d-kube-api-access-65lkc\") pod \"test-curl-7854-02a1ee\" (UID: \"674e1bec-4721-46dc-85ee-3f7399e8e77d\") " pod="opendatahub/test-curl-7854-02a1ee"
Sep 14 11:03:44.533126 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:44.533054    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-02a1ee"
Sep 14 11:03:44.811928 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:03:44.811899    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod674e1bec_4721_46dc_85ee_3f7399e8e77d.slice/crio-3270b47923b68560ef774f10a365017468ae2cd7a7ce148bfc9e837d3b866c4c WatchSource:0}: Error finding container 3270b47923b68560ef774f10a365017468ae2cd7a7ce148bfc9e837d3b866c4c: Status 404 returned error can't find the container with id 3270b47923b68560ef774f10a365017468ae2cd7a7ce148bfc9e837d3b866c4c
Sep 14 11:03:44.812822 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:44.812796    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7854-02a1ee"]
Sep 14 11:03:45.395235 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:45.395192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7854-02a1ee" event={"ID":"674e1bec-4721-46dc-85ee-3f7399e8e77d","Type":"ContainerStarted","Data":"53ee8290aa6a566fd896ed9af0c6048516316d937ae3b485788698d7c5ccd0b6"}
Sep 14 11:03:45.395677 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:45.395244    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7854-02a1ee" event={"ID":"674e1bec-4721-46dc-85ee-3f7399e8e77d","Type":"ContainerStarted","Data":"3270b47923b68560ef774f10a365017468ae2cd7a7ce148bfc9e837d3b866c4c"}
Sep 14 11:03:45.395677 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:45.395241    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7854-4be324" podUID="490aff59-dc96-4120-8084-9bf277402b27" containerName="test-curl-7854-4be324" containerID="cri-o://316824e2dd6fb72a7ed92c1d6bca26491bf7e98be401b7c59f7a2fed4d0a227c" gracePeriod=2
Sep 14 11:03:45.398482 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:03:45.398457    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7854-4be324\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="490aff59-dc96-4120-8084-9bf277402b27" pod="opendatahub/test-curl-7854-4be324"
Sep 14 11:03:45.441040 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:45.440994    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7854-02a1ee" podStartSLOduration=1.440979372 podStartE2EDuration="1.440979372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:45.439736171 +0000 UTC m=+930.124420557" watchObservedRunningTime="2026-09-14 11:03:45.440979372 +0000 UTC m=+930.125663762"
Sep 14 11:03:45.442390 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:03:45.442365    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7854-4be324\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="490aff59-dc96-4120-8084-9bf277402b27" pod="opendatahub/test-curl-7854-4be324"
Sep 14 11:03:45.802067 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:03:45.801965    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7854-4be324\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="490aff59-dc96-4120-8084-9bf277402b27" pod="opendatahub/test-curl-7854-4be324"
Sep 14 11:03:46.093750 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.093662    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7854-02a1ee"]
Sep 14 11:03:46.099455 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.099422    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7854-02a1ee"]
Sep 14 11:03:46.338540 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.338506    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"]
Sep 14 11:03:46.338956 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.338941    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674e1bec-4721-46dc-85ee-3f7399e8e77d" containerName="test-curl-7854-02a1ee"
Sep 14 11:03:46.339013 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.338958    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="674e1bec-4721-46dc-85ee-3f7399e8e77d" containerName="test-curl-7854-02a1ee"
Sep 14 11:03:46.339049 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.339033    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="674e1bec-4721-46dc-85ee-3f7399e8e77d" containerName="test-curl-7854-02a1ee"
Sep 14 11:03:46.341583 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.341565    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:46.344397 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.344328    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-04195c-serving-cert\""
Sep 14 11:03:46.344512 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.344410    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 11:03:46.345561 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.345537    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 11:03:46.345672 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.345546    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksb7s\""
Sep 14 11:03:46.346230 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.346211    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 11:03:46.361029 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.361005    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"]
Sep 14 11:03:46.402325 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:03:46.402298    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7854-02a1ee" container="test-curl-7854-02a1ee"
Sep 14 11:03:46.436811 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.436786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-maas-api-tls\") pod \"maas-api-e2e-inf-a-04195c-6544d95856-rlhgb\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:46.436913 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.436829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hdmk\" (UniqueName: \"kubernetes.io/projected/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-kube-api-access-5hdmk\") pod \"maas-api-e2e-inf-a-04195c-6544d95856-rlhgb\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:46.436913 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.436881    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-metrics-certs\") pod \"maas-api-e2e-inf-a-04195c-6544d95856-rlhgb\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:46.436985 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.436912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04195c-6544d95856-rlhgb\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:46.538373 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.538345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04195c-6544d95856-rlhgb\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:46.538526 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.538441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-maas-api-tls\") pod \"maas-api-e2e-inf-a-04195c-6544d95856-rlhgb\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:46.538526 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.538470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hdmk\" (UniqueName: \"kubernetes.io/projected/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-kube-api-access-5hdmk\") pod \"maas-api-e2e-inf-a-04195c-6544d95856-rlhgb\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:46.538625 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.538541    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-metrics-certs\") pod \"maas-api-e2e-inf-a-04195c-6544d95856-rlhgb\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:46.538625 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:03:46.538594    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-serving-cert: secret "maas-api-e2e-inf-a-04195c-serving-cert" not found
Sep 14 11:03:46.538698 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:03:46.538655    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-maas-api-tls podName:bf7280d4-31a2-4608-89e5-ab9f979c1b0c nodeName:}" failed. No retries permitted until 2026-09-14 11:03:47.038634756 +0000 UTC m=+931.723319122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-maas-api-tls") pod "maas-api-e2e-inf-a-04195c-6544d95856-rlhgb" (UID: "bf7280d4-31a2-4608-89e5-ab9f979c1b0c") : secret "maas-api-e2e-inf-a-04195c-serving-cert" not found
Sep 14 11:03:46.539130 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.539110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04195c-6544d95856-rlhgb\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:46.541005 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.540984    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-metrics-certs\") pod \"maas-api-e2e-inf-a-04195c-6544d95856-rlhgb\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:46.562900 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.562863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hdmk\" (UniqueName: \"kubernetes.io/projected/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-kube-api-access-5hdmk\") pod \"maas-api-e2e-inf-a-04195c-6544d95856-rlhgb\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:46.970249 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.970220    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"]
Sep 14 11:03:46.973920 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.973899    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:46.977566 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.977539    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-b275d0-openshift-default-dockercfg-5hs6d\""
Sep 14 11:03:46.993397 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:46.993372    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"]
Sep 14 11:03:47.043034 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.043000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmzgm\" (UniqueName: \"kubernetes.io/projected/7cbc1301-3425-4ef9-952e-9347fbfae04d-kube-api-access-rmzgm\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.043200 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.043088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-workload-certs\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.043200 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.043177    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-maas-api-tls\") pod \"maas-api-e2e-inf-a-04195c-6544d95856-rlhgb\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:47.043332 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.043245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-envoy\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.043332 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.043275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-podinfo\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.043332 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.043299    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbc1301-3425-4ef9-952e-9347fbfae04d-istiod-ca-cert\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.043332 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.043328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-workload-socket\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.043535 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.043352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-credential-socket\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.043535 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.043416    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-data\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.043535 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.043469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-token\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.046065 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.046042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-maas-api-tls\") pod \"maas-api-e2e-inf-a-04195c-6544d95856-rlhgb\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:47.144373 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.144331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmzgm\" (UniqueName: \"kubernetes.io/projected/7cbc1301-3425-4ef9-952e-9347fbfae04d-kube-api-access-rmzgm\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.144559 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.144401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-workload-certs\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.144559 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.144464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-envoy\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.144559 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.144486    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-podinfo\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.144559 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.144508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbc1301-3425-4ef9-952e-9347fbfae04d-istiod-ca-cert\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.144559 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.144535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-workload-socket\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.144559 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.144558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-credential-socket\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.144981 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.144592    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-data\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.144981 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.144644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-token\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.145083 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.145036    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-workload-certs\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.145336 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.145312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-workload-socket\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.145445 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.145423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-credential-socket\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.145648 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.145629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-data\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.145728 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.145700    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbc1301-3425-4ef9-952e-9347fbfae04d-istiod-ca-cert\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.147498 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.147470    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-envoy\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.148050 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.148027    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-podinfo\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.167812 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.167788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmzgm\" (UniqueName: \"kubernetes.io/projected/7cbc1301-3425-4ef9-952e-9347fbfae04d-kube-api-access-rmzgm\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.171784 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.171756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-token\") pod \"e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") " pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.252888 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.252794    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:47.285780 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.285748    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:47.408997 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.408963    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7854-02a1ee" podUID="674e1bec-4721-46dc-85ee-3f7399e8e77d" containerName="test-curl-7854-02a1ee" containerID="cri-o://53ee8290aa6a566fd896ed9af0c6048516316d937ae3b485788698d7c5ccd0b6" gracePeriod=2
Sep 14 11:03:47.412597 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:03:47.412570    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7854-02a1ee\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="674e1bec-4721-46dc-85ee-3f7399e8e77d" pod="opendatahub/test-curl-7854-02a1ee"
Sep 14 11:03:47.569783 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.569547    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"]
Sep 14 11:03:47.571420 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:03:47.570672    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7280d4_31a2_4608_89e5_ab9f979c1b0c.slice/crio-56dcfeccfc68ce81fb4351e85913ab98e795698934813ee8eafd0aae3a16668e WatchSource:0}: Error finding container 56dcfeccfc68ce81fb4351e85913ab98e795698934813ee8eafd0aae3a16668e: Status 404 returned error can't find the container with id 56dcfeccfc68ce81fb4351e85913ab98e795698934813ee8eafd0aae3a16668e
Sep 14 11:03:47.604383 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:03:47.604341    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbc1301_3425_4ef9_952e_9347fbfae04d.slice/crio-e67fc916a7108b898bee74617e7a409a57172dcb2209a83d0a3d6659f974cd2d WatchSource:0}: Error finding container e67fc916a7108b898bee74617e7a409a57172dcb2209a83d0a3d6659f974cd2d: Status 404 returned error can't find the container with id e67fc916a7108b898bee74617e7a409a57172dcb2209a83d0a3d6659f974cd2d
Sep 14 11:03:47.607541 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.607498    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:03:47.607664 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.607590    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:03:47.607664 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.607635    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:03:47.610389 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.610368    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"]
Sep 14 11:03:47.621313 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.621292    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-4be324"
Sep 14 11:03:47.630960 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:03:47.630926    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7854-02a1ee\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="674e1bec-4721-46dc-85ee-3f7399e8e77d" pod="opendatahub/test-curl-7854-02a1ee"
Sep 14 11:03:47.649561 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.649532    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2phhc\" (UniqueName: \"kubernetes.io/projected/490aff59-dc96-4120-8084-9bf277402b27-kube-api-access-2phhc\") pod \"490aff59-dc96-4120-8084-9bf277402b27\" (UID: \"490aff59-dc96-4120-8084-9bf277402b27\") "
Sep 14 11:03:47.651957 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.651928    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490aff59-dc96-4120-8084-9bf277402b27-kube-api-access-2phhc" (OuterVolumeSpecName: "kube-api-access-2phhc") pod "490aff59-dc96-4120-8084-9bf277402b27" (UID: "490aff59-dc96-4120-8084-9bf277402b27"). InnerVolumeSpecName "kube-api-access-2phhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:03:47.751088 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.751051    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2phhc\" (UniqueName: \"kubernetes.io/projected/490aff59-dc96-4120-8084-9bf277402b27-kube-api-access-2phhc\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:03:47.804182 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:47.804103    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="490aff59-dc96-4120-8084-9bf277402b27" path="/var/lib/kubelet/pods/490aff59-dc96-4120-8084-9bf277402b27/volumes"
Sep 14 11:03:48.416950 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:48.416913    2561 generic.go:356] "Generic (PLEG): container finished" podID="490aff59-dc96-4120-8084-9bf277402b27" containerID="316824e2dd6fb72a7ed92c1d6bca26491bf7e98be401b7c59f7a2fed4d0a227c" exitCode=137
Sep 14 11:03:48.417418 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:48.416967    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-4be324"
Sep 14 11:03:48.417418 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:48.417013    2561 scope.go:117] "RemoveContainer" containerID="316824e2dd6fb72a7ed92c1d6bca26491bf7e98be401b7c59f7a2fed4d0a227c"
Sep 14 11:03:48.419134 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:48.419086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" event={"ID":"7cbc1301-3425-4ef9-952e-9347fbfae04d","Type":"ContainerStarted","Data":"7c879c0b9f26a264466f9cb2cc0eea5e76714c757d45446180ede0ca7a0738aa"}
Sep 14 11:03:48.419248 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:48.419146    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" event={"ID":"7cbc1301-3425-4ef9-952e-9347fbfae04d","Type":"ContainerStarted","Data":"e67fc916a7108b898bee74617e7a409a57172dcb2209a83d0a3d6659f974cd2d"}
Sep 14 11:03:48.420939 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:48.420918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb" event={"ID":"bf7280d4-31a2-4608-89e5-ab9f979c1b0c","Type":"ContainerStarted","Data":"56dcfeccfc68ce81fb4351e85913ab98e795698934813ee8eafd0aae3a16668e"}
Sep 14 11:03:48.426385 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:48.426367    2561 scope.go:117] "RemoveContainer" containerID="316824e2dd6fb72a7ed92c1d6bca26491bf7e98be401b7c59f7a2fed4d0a227c"
Sep 14 11:03:48.426632 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:03:48.426612    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316824e2dd6fb72a7ed92c1d6bca26491bf7e98be401b7c59f7a2fed4d0a227c\": container with ID starting with 316824e2dd6fb72a7ed92c1d6bca26491bf7e98be401b7c59f7a2fed4d0a227c not found: ID does not exist" containerID="316824e2dd6fb72a7ed92c1d6bca26491bf7e98be401b7c59f7a2fed4d0a227c"
Sep 14 11:03:48.426693 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:48.426643    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316824e2dd6fb72a7ed92c1d6bca26491bf7e98be401b7c59f7a2fed4d0a227c"} err="failed to get container status \"316824e2dd6fb72a7ed92c1d6bca26491bf7e98be401b7c59f7a2fed4d0a227c\": rpc error: code = NotFound desc = could not find container \"316824e2dd6fb72a7ed92c1d6bca26491bf7e98be401b7c59f7a2fed4d0a227c\": container with ID starting with 316824e2dd6fb72a7ed92c1d6bca26491bf7e98be401b7c59f7a2fed4d0a227c not found: ID does not exist"
Sep 14 11:03:48.443896 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:48.443850    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" podStartSLOduration=2.443835147 podStartE2EDuration="2.443835147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:48.44151517 +0000 UTC m=+933.126199571" watchObservedRunningTime="2026-09-14 11:03:48.443835147 +0000 UTC m=+933.128519606"
Sep 14 11:03:49.286049 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:49.286013    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:49.291863 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:49.291842    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:49.433294 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:49.433266    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:49.434467 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:49.434444    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:03:50.179490 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.179462    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"]
Sep 14 11:03:50.183760 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.183742    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.187244 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.187051    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6706857a-openshift-default-dockercfg-h2gbq\""
Sep 14 11:03:50.199798 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.199771    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"]
Sep 14 11:03:50.281373 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.281331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-workload-socket\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.281661 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.281641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-workload-certs\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.281862 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.281841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-credential-socket\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.282068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.282053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-token\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.282228 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.282213    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd9c2\" (UniqueName: \"kubernetes.io/projected/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-kube-api-access-rd9c2\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.282384 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.282369    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-data\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.282539 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.282523    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istiod-ca-cert\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.282806 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.282788    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-envoy\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.283013 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.282996    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-podinfo\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.385068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.385021    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-envoy\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.385068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.385064    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-podinfo\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.385324 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.385256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-workload-socket\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.385324 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.385317    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-workload-certs\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.385448 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.385347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-credential-socket\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.385448 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.385376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-token\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.385448 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.385402    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd9c2\" (UniqueName: \"kubernetes.io/projected/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-kube-api-access-rd9c2\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.385448 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.385430    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-data\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.385650 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.385465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istiod-ca-cert\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.385650 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.385603    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-workload-socket\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.385990 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.385964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-credential-socket\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.386241 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.386220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-data\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.386340 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.386311    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istiod-ca-cert\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.386433 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.386409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-workload-certs\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.387696 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.387674    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-envoy\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.388438 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.388416    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-podinfo\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.408768 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.408681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-token\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.408768 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.408758    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd9c2\" (UniqueName: \"kubernetes.io/projected/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-kube-api-access-rd9c2\") pod \"e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") " pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.441794 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.441763    2561 generic.go:356] "Generic (PLEG): container finished" podID="674e1bec-4721-46dc-85ee-3f7399e8e77d" containerID="53ee8290aa6a566fd896ed9af0c6048516316d937ae3b485788698d7c5ccd0b6" exitCode=137
Sep 14 11:03:50.497657 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.497625    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:50.625572 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.625541    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-02a1ee"
Sep 14 11:03:50.689964 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.689942    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65lkc\" (UniqueName: \"kubernetes.io/projected/674e1bec-4721-46dc-85ee-3f7399e8e77d-kube-api-access-65lkc\") pod \"674e1bec-4721-46dc-85ee-3f7399e8e77d\" (UID: \"674e1bec-4721-46dc-85ee-3f7399e8e77d\") "
Sep 14 11:03:50.692043 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.692011    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674e1bec-4721-46dc-85ee-3f7399e8e77d-kube-api-access-65lkc" (OuterVolumeSpecName: "kube-api-access-65lkc") pod "674e1bec-4721-46dc-85ee-3f7399e8e77d" (UID: "674e1bec-4721-46dc-85ee-3f7399e8e77d"). InnerVolumeSpecName "kube-api-access-65lkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:03:50.791782 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.791751    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65lkc\" (UniqueName: \"kubernetes.io/projected/674e1bec-4721-46dc-85ee-3f7399e8e77d-kube-api-access-65lkc\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:03:50.904249 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.904220    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"]
Sep 14 11:03:50.905215 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:03:50.905189    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b6e067_afb7_4c3f_aa4d_44a836aff3ae.slice/crio-785173802b9f760914a792e7b0f755588d03abec3cc9590487a875b7409229b5 WatchSource:0}: Error finding container 785173802b9f760914a792e7b0f755588d03abec3cc9590487a875b7409229b5: Status 404 returned error can't find the container with id 785173802b9f760914a792e7b0f755588d03abec3cc9590487a875b7409229b5
Sep 14 11:03:50.907628 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.907592    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:03:50.907698 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.907669    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:03:50.907765 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:50.907696    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:03:51.453376 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:51.453337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" event={"ID":"30b6e067-afb7-4c3f-aa4d-44a836aff3ae","Type":"ContainerStarted","Data":"a7e386ce7bf8b7365fa64d9157fe76e457b288d0e3058a7f82d81a7e64bec1af"}
Sep 14 11:03:51.453887 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:51.453407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" event={"ID":"30b6e067-afb7-4c3f-aa4d-44a836aff3ae","Type":"ContainerStarted","Data":"785173802b9f760914a792e7b0f755588d03abec3cc9590487a875b7409229b5"}
Sep 14 11:03:51.454474 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:51.454447    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-02a1ee"
Sep 14 11:03:51.475917 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:51.475858    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb" event={"ID":"bf7280d4-31a2-4608-89e5-ab9f979c1b0c","Type":"ContainerStarted","Data":"8de616025b932726d94564263c3501acc4f0c36beb6893b054500482160467c9"}
Sep 14 11:03:51.476138 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:51.476002    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:03:51.476138 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:51.476071    2561 scope.go:117] "RemoveContainer" containerID="53ee8290aa6a566fd896ed9af0c6048516316d937ae3b485788698d7c5ccd0b6"
Sep 14 11:03:51.497880 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:51.497851    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:51.502754 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:51.502693    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" podStartSLOduration=1.5026785390000001 podStartE2EDuration="1.502678539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:51.500169676 +0000 UTC m=+936.184854088" watchObservedRunningTime="2026-09-14 11:03:51.502678539 +0000 UTC m=+936.187362925"
Sep 14 11:03:51.504820 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:51.504801    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:51.524308 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:51.524267    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb" podStartSLOduration=2.411022782 podStartE2EDuration="5.524254179s" totalImagesPullingTime="3.113231397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:51.523374029 +0000 UTC m=+936.208058442" watchObservedRunningTime="2026-09-14 11:03:51.524254179 +0000 UTC m=+936.208938568"
Sep 14 11:03:51.525615 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:03:51.525584    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7854-02a1ee\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="674e1bec-4721-46dc-85ee-3f7399e8e77d" pod="opendatahub/test-curl-7854-02a1ee"
Sep 14 11:03:51.560281 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:03:51.560238    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7854-02a1ee\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="674e1bec-4721-46dc-85ee-3f7399e8e77d" pod="opendatahub/test-curl-7854-02a1ee"
Sep 14 11:03:51.807014 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:51.806942    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="674e1bec-4721-46dc-85ee-3f7399e8e77d" path="/var/lib/kubelet/pods/674e1bec-4721-46dc-85ee-3f7399e8e77d/volumes"
Sep 14 11:03:52.494209 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:52.494181    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:52.498417 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:52.498396    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:03:57.500766 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:03:57.500705    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:04:00.550589 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:00.550500    2561 generic.go:356] "Generic (PLEG): container finished" podID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerID="44c6aecdf266989e29021f14b68dfe1e0a8aac4c9f2d52e46af12ac452adb2d1" exitCode=0
Sep 14 11:04:00.551115 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:00.550580    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" event={"ID":"f51381b7-acf4-4af2-b15a-1dc7119d2c6c","Type":"ContainerDied","Data":"44c6aecdf266989e29021f14b68dfe1e0a8aac4c9f2d52e46af12ac452adb2d1"}
Sep 14 11:04:00.551190 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:00.551134    2561 scope.go:117] "RemoveContainer" containerID="44c6aecdf266989e29021f14b68dfe1e0a8aac4c9f2d52e46af12ac452adb2d1"
Sep 14 11:04:00.552592 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:00.552558    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:00.552697 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:00.552651    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:00.552783 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:00.552696    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:01.563333 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:01.563289    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" event={"ID":"f51381b7-acf4-4af2-b15a-1dc7119d2c6c","Type":"ContainerStarted","Data":"5f84a30ba15cbde4d71251dc0cfb75d10bea2f28aa7693556a2b83775bb045ac"}
Sep 14 11:04:01.567023 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:01.566985    2561 generic.go:356] "Generic (PLEG): container finished" podID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerID="4522dcc307674370c5d536d58e1c68c8de1d0c9b47e810d362c886096f377d12" exitCode=0
Sep 14 11:04:01.567180 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:01.567111    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" event={"ID":"0d6291a6-7a72-48f0-bf8c-80dd9ed61138","Type":"ContainerDied","Data":"4522dcc307674370c5d536d58e1c68c8de1d0c9b47e810d362c886096f377d12"}
Sep 14 11:04:01.567615 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:01.567595    2561 scope.go:117] "RemoveContainer" containerID="4522dcc307674370c5d536d58e1c68c8de1d0c9b47e810d362c886096f377d12"
Sep 14 11:04:01.569205 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:01.569168    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:01.569305 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:01.569257    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:01.569369 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:01.569308    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:02.430595 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:02.430563    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:04:02.430879 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:02.430858    2561 patch_prober.go:28] interesting pod/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 14 11:04:02.430948 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:02.430919    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 14 11:04:02.581911 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:02.581880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" event={"ID":"0d6291a6-7a72-48f0-bf8c-80dd9ed61138","Type":"ContainerStarted","Data":"d5365beae84a8cacf12b797ae21a6e262ab392bcc83440205581305eaef7c4c1"}
Sep 14 11:04:02.932556 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:02.932515    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"]
Sep 14 11:04:02.937058 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:02.937036    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:02.962067 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:02.948986    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0777c1-openshift-default-dockercfg-t45cg\""
Sep 14 11:04:02.980805 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:02.980779    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"]
Sep 14 11:04:03.019106 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.019023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-envoy\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.019106 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.019065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-workload-certs\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.019345 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.019149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-podinfo\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.019345 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.019244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-workload-socket\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.019345 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.019263    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-data\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.019345 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.019286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-token\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.019559 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.019362    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istiod-ca-cert\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.019559 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.019392    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-credential-socket\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.019559 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.019436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfbhv\" (UniqueName: \"kubernetes.io/projected/af96dc32-00b9-4d16-8303-3a4ddb5cb146-kube-api-access-nfbhv\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.121259 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.120788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-workload-socket\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.121458 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.121429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-data\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.121539 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.121469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-token\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.121597 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.121546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istiod-ca-cert\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.121597 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.121586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-credential-socket\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.121703 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.121641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfbhv\" (UniqueName: \"kubernetes.io/projected/af96dc32-00b9-4d16-8303-3a4ddb5cb146-kube-api-access-nfbhv\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.121703 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.121674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-envoy\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.121836 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.121702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-workload-certs\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.121836 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.121784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-podinfo\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.122420 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.122393    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istiod-ca-cert\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.122884 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.122858    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-data\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.123032 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.122877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-workload-certs\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.123121 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.121232    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-workload-socket\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.123280 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.123259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-credential-socket\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.136741 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.136224    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-podinfo\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.136741 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.136444    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-envoy\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.139222 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.139192    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-token\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.155503 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.155467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfbhv\" (UniqueName: \"kubernetes.io/projected/af96dc32-00b9-4d16-8303-3a4ddb5cb146-kube-api-access-nfbhv\") pod \"e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") " pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.260280 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.260170    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:03.431735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.431683    2561 patch_prober.go:28] interesting pod/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 14 11:04:03.431921 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.431770    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 14 11:04:03.516419 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.516331    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:04:03.516419 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.516374    2561 patch_prober.go:28] interesting pod/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 14 11:04:03.516604 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.516421    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 14 11:04:03.611774 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.611745    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"]
Sep 14 11:04:03.612559 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:04:03.612533    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf96dc32_00b9_4d16_8303_3a4ddb5cb146.slice/crio-6dd7a9264e506212090e818b8e3697d9576f4576d6abfb97a3a770fb883a8b22 WatchSource:0}: Error finding container 6dd7a9264e506212090e818b8e3697d9576f4576d6abfb97a3a770fb883a8b22: Status 404 returned error can't find the container with id 6dd7a9264e506212090e818b8e3697d9576f4576d6abfb97a3a770fb883a8b22
Sep 14 11:04:03.615191 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.615161    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:03.615262 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.615221    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:03.615262 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.615249    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:03.754907 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:03.754876    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:04:04.430923 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:04.430894    2561 patch_prober.go:28] interesting pod/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 14 11:04:04.431141 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:04.431119    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 14 11:04:04.516140 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:04.516107    2561 patch_prober.go:28] interesting pod/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 14 11:04:04.516309 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:04.516167    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 14 11:04:04.604319 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:04.604283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p" event={"ID":"af96dc32-00b9-4d16-8303-3a4ddb5cb146","Type":"ContainerStarted","Data":"2de697a2e8020e08f8f6577bc77d62c1822a1c9991eed06584796ff02e709566"}
Sep 14 11:04:04.604502 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:04.604337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p" event={"ID":"af96dc32-00b9-4d16-8303-3a4ddb5cb146","Type":"ContainerStarted","Data":"6dd7a9264e506212090e818b8e3697d9576f4576d6abfb97a3a770fb883a8b22"}
Sep 14 11:04:04.632625 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:04.632584    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p" podStartSLOduration=2.632569252 podStartE2EDuration="2.632569252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:04:04.630644481 +0000 UTC m=+949.315328868" watchObservedRunningTime="2026-09-14 11:04:04.632569252 +0000 UTC m=+949.317253631"
Sep 14 11:04:05.261577 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:05.261541    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:05.272262 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:05.272235    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:05.431466 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:05.431434    2561 patch_prober.go:28] interesting pod/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 14 11:04:05.431630 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:05.431483    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 14 11:04:05.516684 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:05.516585    2561 patch_prober.go:28] interesting pod/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 14 11:04:05.516684 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:05.516648    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 14 11:04:05.611666 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:05.611610    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:05.612468 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:05.612444    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:06.430654 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:06.430620    2561 patch_prober.go:28] interesting pod/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 14 11:04:06.431068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:06.430673    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 14 11:04:06.516476 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:06.516438    2561 patch_prober.go:28] interesting pod/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 14 11:04:06.516682 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:06.516512    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 14 11:04:07.435371 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:07.435337    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:04:07.436075 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:07.436012    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:04:07.436896 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:07.436878    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:04:07.516908 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:07.516873    2561 patch_prober.go:28] interesting pod/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 14 11:04:07.517175 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:07.517144    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 14 11:04:08.516690 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:08.516651    2561 patch_prober.go:28] interesting pod/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 14 11:04:08.517296 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:08.517262    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 14 11:04:09.521080 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:09.521048    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:04:09.521445 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:09.521356    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:04:09.522244 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:09.522227    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:04:11.163735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.159375    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"]
Sep 14 11:04:11.181319 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.181295    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:11.192767 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.192081    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0777c1-serving-cert\""
Sep 14 11:04:11.211938 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.211371    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"]
Sep 14 11:04:11.321663 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.321448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-maas-api-tls\") pod \"maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:11.321663 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.321496    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx699\" (UniqueName: \"kubernetes.io/projected/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-kube-api-access-kx699\") pod \"maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:11.321663 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.321527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-metrics-certs\") pod \"maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:11.321663 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.321548    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:11.422589 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.422503    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx699\" (UniqueName: \"kubernetes.io/projected/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-kube-api-access-kx699\") pod \"maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:11.422909 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.422885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-metrics-certs\") pod \"maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:11.423092 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.423074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:11.423854 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.423835    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:11.423993 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.423919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-maas-api-tls\") pod \"maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:11.424132 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:04:11.424048    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-serving-cert: secret "maas-api-e2e-inf-b-0777c1-serving-cert" not found
Sep 14 11:04:11.424249 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:04:11.424236    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-maas-api-tls podName:3a1cf7e5-47bf-4845-9909-d0f45bb7d891 nodeName:}" failed. No retries permitted until 2026-09-14 11:04:11.924214788 +0000 UTC m=+956.608899158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-maas-api-tls") pod "maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7" (UID: "3a1cf7e5-47bf-4845-9909-d0f45bb7d891") : secret "maas-api-e2e-inf-b-0777c1-serving-cert" not found
Sep 14 11:04:11.425657 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.425584    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-metrics-certs\") pod \"maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:11.441942 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.441922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx699\" (UniqueName: \"kubernetes.io/projected/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-kube-api-access-kx699\") pod \"maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:11.932499 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.932454    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-maas-api-tls\") pod \"maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:11.934813 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:11.934791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-maas-api-tls\") pod \"maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:12.118444 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:12.118402    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:12.425912 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:12.425884    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"]
Sep 14 11:04:12.426945 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:04:12.426913    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1cf7e5_47bf_4845_9909_d0f45bb7d891.slice/crio-8dfcd975568f8304bd11d7da8240a983c42c60c1272f6fa0e908a39bc95c0c03 WatchSource:0}: Error finding container 8dfcd975568f8304bd11d7da8240a983c42c60c1272f6fa0e908a39bc95c0c03: Status 404 returned error can't find the container with id 8dfcd975568f8304bd11d7da8240a983c42c60c1272f6fa0e908a39bc95c0c03
Sep 14 11:04:12.669284 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:12.669254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7" event={"ID":"3a1cf7e5-47bf-4845-9909-d0f45bb7d891","Type":"ContainerStarted","Data":"8dfcd975568f8304bd11d7da8240a983c42c60c1272f6fa0e908a39bc95c0c03"}
Sep 14 11:04:13.676765 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:13.676664    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7" event={"ID":"3a1cf7e5-47bf-4845-9909-d0f45bb7d891","Type":"ContainerStarted","Data":"50bc05f22777702098903ed0fb8b29e1f88de304422d9253639cf71e73a59ece"}
Sep 14 11:04:13.677218 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:13.676889    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:13.697059 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:13.696993    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7" podStartSLOduration=2.420952661 podStartE2EDuration="2.696970549s" totalImagesPullingTime="276.017888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:04:13.695005827 +0000 UTC m=+958.379690217" watchObservedRunningTime="2026-09-14 11:04:13.696970549 +0000 UTC m=+958.381654939"
Sep 14 11:04:15.690660 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:15.690580    2561 generic.go:356] "Generic (PLEG): container finished" podID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerID="a7e386ce7bf8b7365fa64d9157fe76e457b288d0e3058a7f82d81a7e64bec1af" exitCode=0
Sep 14 11:04:15.691169 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:15.690659    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" event={"ID":"30b6e067-afb7-4c3f-aa4d-44a836aff3ae","Type":"ContainerDied","Data":"a7e386ce7bf8b7365fa64d9157fe76e457b288d0e3058a7f82d81a7e64bec1af"}
Sep 14 11:04:15.691235 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:15.691210    2561 scope.go:117] "RemoveContainer" containerID="a7e386ce7bf8b7365fa64d9157fe76e457b288d0e3058a7f82d81a7e64bec1af"
Sep 14 11:04:15.692530 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:15.692495    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:15.692659 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:15.692566    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:15.692659 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:15.692583    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" event={"ID":"7cbc1301-3425-4ef9-952e-9347fbfae04d","Type":"ContainerDied","Data":"7c879c0b9f26a264466f9cb2cc0eea5e76714c757d45446180ede0ca7a0738aa"}
Sep 14 11:04:15.692659 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:15.692605    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:15.692789 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:15.692569    2561 generic.go:356] "Generic (PLEG): container finished" podID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerID="7c879c0b9f26a264466f9cb2cc0eea5e76714c757d45446180ede0ca7a0738aa" exitCode=0
Sep 14 11:04:15.692932 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:15.692919    2561 scope.go:117] "RemoveContainer" containerID="7c879c0b9f26a264466f9cb2cc0eea5e76714c757d45446180ede0ca7a0738aa"
Sep 14 11:04:15.694040 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:15.693956    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:15.694040 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:15.694015    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:15.694124 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:15.694046    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:16.700415 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:16.700376    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" event={"ID":"7cbc1301-3425-4ef9-952e-9347fbfae04d","Type":"ContainerStarted","Data":"f018ea5dd4928fc5df2fcba11bd67d86186fce42d9cbcd111795fcd6156bf2a8"}
Sep 14 11:04:16.702412 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:16.702390    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" event={"ID":"30b6e067-afb7-4c3f-aa4d-44a836aff3ae","Type":"ContainerStarted","Data":"d4cdbe366d29b1d828ba22d102748d6457e3027c122e45f29fe409fbf69e284d"}
Sep 14 11:04:17.285977 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:17.285933    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:04:17.287460 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:17.287423    2561 patch_prober.go:28] interesting pod/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 14 11:04:17.287661 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:17.287512    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 14 11:04:17.497882 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:17.497845    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:04:17.498266 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:17.498241    2561 patch_prober.go:28] interesting pod/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 14 11:04:17.498365 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:17.498309    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 14 11:04:18.286813 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.286772    2561 patch_prober.go:28] interesting pod/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 14 11:04:18.287306 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.286853    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 14 11:04:18.333083 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.333048    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"]
Sep 14 11:04:18.337957 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.337937    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.341606 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.341586    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"kube-root-ca.crt\""
Sep 14 11:04:18.342511 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.342494    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"openshift-service-ca.crt\""
Sep 14 11:04:18.342652 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.342639    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"simulator-w4-4ebf73-kserve-self-signed-certs\""
Sep 14 11:04:18.342784 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.342767    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"default-dockercfg-8975n\""
Sep 14 11:04:18.356863 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.356838    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"]
Sep 14 11:04:18.498488 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.498454    2561 patch_prober.go:28] interesting pod/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 14 11:04:18.498799 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.498758    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 14 11:04:18.505558 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.505536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a474c605-06a5-418c-a248-6236c629c6e9-tls-certs\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.505649 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.505572    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-model-cache\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.505649 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.505642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-home\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.505735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.505672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-dshm\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.505780 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.505761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz66v\" (UniqueName: \"kubernetes.io/projected/a474c605-06a5-418c-a248-6236c629c6e9-kube-api-access-fz66v\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.505824 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.505809    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-tmp-dir\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.606352 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.606325    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz66v\" (UniqueName: \"kubernetes.io/projected/a474c605-06a5-418c-a248-6236c629c6e9-kube-api-access-fz66v\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.606567 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.606551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-tmp-dir\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.606674 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.606665    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a474c605-06a5-418c-a248-6236c629c6e9-tls-certs\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.606810 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.606790    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-model-cache\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.606886 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.606870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-home\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.606942 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.606924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-dshm\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.607165 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.607136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-tmp-dir\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.607271 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.607185    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-home\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.607271 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.607264    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-model-cache\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.609111 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.609078    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-dshm\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.609111 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.609082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a474c605-06a5-418c-a248-6236c629c6e9-tls-certs\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.616330 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.616308    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz66v\" (UniqueName: \"kubernetes.io/projected/a474c605-06a5-418c-a248-6236c629c6e9-kube-api-access-fz66v\") pod \"simulator-w4-4ebf73-kserve-6d98946478-ml962\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") " pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.653209 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.653181    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:18.968970 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:18.968855    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"]
Sep 14 11:04:18.977410 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:04:18.977376    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda474c605_06a5_418c_a248_6236c629c6e9.slice/crio-7b25d58063a73618fc1c23302860d13f69300398aa86cff51897f7b8d222c64e WatchSource:0}: Error finding container 7b25d58063a73618fc1c23302860d13f69300398aa86cff51897f7b8d222c64e: Status 404 returned error can't find the container with id 7b25d58063a73618fc1c23302860d13f69300398aa86cff51897f7b8d222c64e
Sep 14 11:04:19.287794 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:19.287280    2561 patch_prober.go:28] interesting pod/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 14 11:04:19.287794 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:19.287344    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 14 11:04:19.433260 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:19.433223    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:04:19.500638 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:19.500183    2561 patch_prober.go:28] interesting pod/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 14 11:04:19.500638 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:19.500245    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 14 11:04:19.701742 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:19.700969    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:04:19.732737 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:19.729325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962" event={"ID":"a474c605-06a5-418c-a248-6236c629c6e9","Type":"ContainerStarted","Data":"7b25d58063a73618fc1c23302860d13f69300398aa86cff51897f7b8d222c64e"}
Sep 14 11:04:20.287259 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:20.286777    2561 patch_prober.go:28] interesting pod/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 14 11:04:20.287259 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:20.286848    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 14 11:04:20.498981 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:20.498935    2561 patch_prober.go:28] interesting pod/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 14 11:04:20.499471 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:20.499005    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 14 11:04:21.288206 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:21.288166    2561 patch_prober.go:28] interesting pod/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 14 11:04:21.288946 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:21.288488    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 14 11:04:21.498242 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:21.498197    2561 patch_prober.go:28] interesting pod/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 14 11:04:21.498451 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:21.498268    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 14 11:04:22.287206 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:22.286756    2561 patch_prober.go:28] interesting pod/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 14 11:04:22.287206 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:22.286817    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 14 11:04:22.495039 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:22.494927    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:04:22.498465 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:22.498053    2561 patch_prober.go:28] interesting pod/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 14 11:04:22.498465 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:22.498106    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 14 11:04:22.757253 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:22.757160    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962" event={"ID":"a474c605-06a5-418c-a248-6236c629c6e9","Type":"ContainerStarted","Data":"2ce13cbfd7ab8900d8bcd8cb23601887e8e4c5c5387070054a837d2641f5681a"}
Sep 14 11:04:23.291030 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:23.291000    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:04:23.291948 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:23.291925    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:04:23.315375 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:23.315328    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962" podStartSLOduration=2.0891125 podStartE2EDuration="5.315310031s" totalImagesPullingTime="3.226197531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:04:22.787812848 +0000 UTC m=+967.472497227" watchObservedRunningTime="2026-09-14 11:04:23.315310031 +0000 UTC m=+967.999994477"
Sep 14 11:04:23.502227 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:23.502194    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:04:23.503225 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:23.503202    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:04:26.726959 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:26.726924    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:04:28.653693 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:28.653664    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:28.654222 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:28.653739    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:28.665828 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:28.665803    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:28.803257 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:28.803222    2561 generic.go:356] "Generic (PLEG): container finished" podID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerID="2de697a2e8020e08f8f6577bc77d62c1822a1c9991eed06584796ff02e709566" exitCode=0
Sep 14 11:04:28.803446 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:28.803289    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p" event={"ID":"af96dc32-00b9-4d16-8303-3a4ddb5cb146","Type":"ContainerDied","Data":"2de697a2e8020e08f8f6577bc77d62c1822a1c9991eed06584796ff02e709566"}
Sep 14 11:04:28.803781 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:28.803760    2561 scope.go:117] "RemoveContainer" containerID="2de697a2e8020e08f8f6577bc77d62c1822a1c9991eed06584796ff02e709566"
Sep 14 11:04:28.805213 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:28.805182    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:28.805289 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:28.805265    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:28.805330 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:28.805307    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:04:28.815213 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:28.815194    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:04:29.813774 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:29.813738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p" event={"ID":"af96dc32-00b9-4d16-8303-3a4ddb5cb146","Type":"ContainerStarted","Data":"cbed4184e7f96a5c0ee297ee4e447c18a613eb98d3f979909018287ebdcde9fd"}
Sep 14 11:04:30.261606 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:30.261563    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:30.261865 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:30.261833    2561 patch_prober.go:28] interesting pod/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 14 11:04:30.261955 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:30.261905    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p" podUID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 14 11:04:31.264974 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:31.264937    2561 patch_prober.go:28] interesting pod/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 14 11:04:31.265523 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:31.265492    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p" podUID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 14 11:04:32.261886 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:32.261845    2561 patch_prober.go:28] interesting pod/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 14 11:04:32.262168 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:32.262138    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p" podUID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 14 11:04:33.261211 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:33.261169    2561 patch_prober.go:28] interesting pod/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 14 11:04:33.261737 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:33.261262    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p" podUID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 14 11:04:34.261368 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:34.261332    2561 patch_prober.go:28] interesting pod/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 14 11:04:34.261992 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:34.261941    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p" podUID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 14 11:04:34.355736 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:34.354610    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"]
Sep 14 11:04:34.355736 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:34.354877    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy" containerID="cri-o://d4cdbe366d29b1d828ba22d102748d6457e3027c122e45f29fe409fbf69e284d" gracePeriod=30
Sep 14 11:04:35.264911 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:35.264881    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:35.265375 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:35.265355    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:35.265967 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:35.265944    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:04:38.504959 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:38.504867    2561 patch_prober.go:28] interesting pod/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 14 11:04:38.504959 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:38.504937    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 11:04:39.608979 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.608951    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:04:39.682662 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.682594    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-workload-socket\") pod \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") "
Sep 14 11:04:39.682662 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.682640    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-workload-certs\") pod \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") "
Sep 14 11:04:39.682851 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.682670    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-podinfo\") pod \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") "
Sep 14 11:04:39.682851 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.682703    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-credential-socket\") pod \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") "
Sep 14 11:04:39.682851 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.682778    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-data\") pod \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") "
Sep 14 11:04:39.682851 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.682827    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-token\") pod \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") "
Sep 14 11:04:39.683068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.682913    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istiod-ca-cert\") pod \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") "
Sep 14 11:04:39.683068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.682929    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30b6e067-afb7-4c3f-aa4d-44a836aff3ae" (UID: "30b6e067-afb7-4c3f-aa4d-44a836aff3ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:04:39.683068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.682946    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd9c2\" (UniqueName: \"kubernetes.io/projected/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-kube-api-access-rd9c2\") pod \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") "
Sep 14 11:04:39.683068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.682990    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-envoy\") pod \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\" (UID: \"30b6e067-afb7-4c3f-aa4d-44a836aff3ae\") "
Sep 14 11:04:39.683278 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.682965    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30b6e067-afb7-4c3f-aa4d-44a836aff3ae" (UID: "30b6e067-afb7-4c3f-aa4d-44a836aff3ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:04:39.683278 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.683018    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30b6e067-afb7-4c3f-aa4d-44a836aff3ae" (UID: "30b6e067-afb7-4c3f-aa4d-44a836aff3ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:04:39.683278 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.683141    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-data" (OuterVolumeSpecName: "istio-data") pod "30b6e067-afb7-4c3f-aa4d-44a836aff3ae" (UID: "30b6e067-afb7-4c3f-aa4d-44a836aff3ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:04:39.683278 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.683262    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30b6e067-afb7-4c3f-aa4d-44a836aff3ae" (UID: "30b6e067-afb7-4c3f-aa4d-44a836aff3ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:04:39.683446 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.683377    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-data\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:04:39.683446 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.683392    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istiod-ca-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:04:39.683446 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.683405    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-workload-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:04:39.683446 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.683418    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-workload-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:04:39.683446 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.683440    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-credential-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:04:39.685543 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.685492    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30b6e067-afb7-4c3f-aa4d-44a836aff3ae" (UID: "30b6e067-afb7-4c3f-aa4d-44a836aff3ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:04:39.685543 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.685486    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30b6e067-afb7-4c3f-aa4d-44a836aff3ae" (UID: "30b6e067-afb7-4c3f-aa4d-44a836aff3ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 11:04:39.685745 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.685611    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-token" (OuterVolumeSpecName: "istio-token") pod "30b6e067-afb7-4c3f-aa4d-44a836aff3ae" (UID: "30b6e067-afb7-4c3f-aa4d-44a836aff3ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:04:39.685745 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.685642    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-kube-api-access-rd9c2" (OuterVolumeSpecName: "kube-api-access-rd9c2") pod "30b6e067-afb7-4c3f-aa4d-44a836aff3ae" (UID: "30b6e067-afb7-4c3f-aa4d-44a836aff3ae"). InnerVolumeSpecName "kube-api-access-rd9c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:04:39.784004 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.783971    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-podinfo\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:04:39.784004 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.784002    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:04:39.784004 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.784011    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd9c2\" (UniqueName: \"kubernetes.io/projected/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-kube-api-access-rd9c2\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:04:39.784243 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.784020    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b6e067-afb7-4c3f-aa4d-44a836aff3ae-istio-envoy\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:04:39.874932 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.874903    2561 generic.go:356] "Generic (PLEG): container finished" podID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerID="d4cdbe366d29b1d828ba22d102748d6457e3027c122e45f29fe409fbf69e284d" exitCode=0
Sep 14 11:04:39.875081 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.874941    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" event={"ID":"30b6e067-afb7-4c3f-aa4d-44a836aff3ae","Type":"ContainerDied","Data":"d4cdbe366d29b1d828ba22d102748d6457e3027c122e45f29fe409fbf69e284d"}
Sep 14 11:04:39.875081 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.874961    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"
Sep 14 11:04:39.875081 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.875003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98" event={"ID":"30b6e067-afb7-4c3f-aa4d-44a836aff3ae","Type":"ContainerDied","Data":"785173802b9f760914a792e7b0f755588d03abec3cc9590487a875b7409229b5"}
Sep 14 11:04:39.875081 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.875027    2561 scope.go:117] "RemoveContainer" containerID="d4cdbe366d29b1d828ba22d102748d6457e3027c122e45f29fe409fbf69e284d"
Sep 14 11:04:39.884392 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.884375    2561 scope.go:117] "RemoveContainer" containerID="a7e386ce7bf8b7365fa64d9157fe76e457b288d0e3058a7f82d81a7e64bec1af"
Sep 14 11:04:39.896554 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.896532    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"]
Sep 14 11:04:39.898675 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.898656    2561 scope.go:117] "RemoveContainer" containerID="d4cdbe366d29b1d828ba22d102748d6457e3027c122e45f29fe409fbf69e284d"
Sep 14 11:04:39.898990 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:04:39.898968    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4cdbe366d29b1d828ba22d102748d6457e3027c122e45f29fe409fbf69e284d\": container with ID starting with d4cdbe366d29b1d828ba22d102748d6457e3027c122e45f29fe409fbf69e284d not found: ID does not exist" containerID="d4cdbe366d29b1d828ba22d102748d6457e3027c122e45f29fe409fbf69e284d"
Sep 14 11:04:39.899061 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.898992    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4cdbe366d29b1d828ba22d102748d6457e3027c122e45f29fe409fbf69e284d"} err="failed to get container status \"d4cdbe366d29b1d828ba22d102748d6457e3027c122e45f29fe409fbf69e284d\": rpc error: code = NotFound desc = could not find container \"d4cdbe366d29b1d828ba22d102748d6457e3027c122e45f29fe409fbf69e284d\": container with ID starting with d4cdbe366d29b1d828ba22d102748d6457e3027c122e45f29fe409fbf69e284d not found: ID does not exist"
Sep 14 11:04:39.899121 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.899064    2561 scope.go:117] "RemoveContainer" containerID="a7e386ce7bf8b7365fa64d9157fe76e457b288d0e3058a7f82d81a7e64bec1af"
Sep 14 11:04:39.899401 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:04:39.899374    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e386ce7bf8b7365fa64d9157fe76e457b288d0e3058a7f82d81a7e64bec1af\": container with ID starting with a7e386ce7bf8b7365fa64d9157fe76e457b288d0e3058a7f82d81a7e64bec1af not found: ID does not exist" containerID="a7e386ce7bf8b7365fa64d9157fe76e457b288d0e3058a7f82d81a7e64bec1af"
Sep 14 11:04:39.899513 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.899407    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e386ce7bf8b7365fa64d9157fe76e457b288d0e3058a7f82d81a7e64bec1af"} err="failed to get container status \"a7e386ce7bf8b7365fa64d9157fe76e457b288d0e3058a7f82d81a7e64bec1af\": rpc error: code = NotFound desc = could not find container \"a7e386ce7bf8b7365fa64d9157fe76e457b288d0e3058a7f82d81a7e64bec1af\": container with ID starting with a7e386ce7bf8b7365fa64d9157fe76e457b288d0e3058a7f82d81a7e64bec1af not found: ID does not exist"
Sep 14 11:04:39.903065 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:39.903046    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6706857a-openshift-default-7b98fbbdf6-fxm98"]
Sep 14 11:04:41.158699 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.158666    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"]
Sep 14 11:04:41.159345 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.159323    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy"
Sep 14 11:04:41.159439 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.159348    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy"
Sep 14 11:04:41.159439 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.159371    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy"
Sep 14 11:04:41.159439 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.159379    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy"
Sep 14 11:04:41.159591 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.159505    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy"
Sep 14 11:04:41.159591 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.159519    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" containerName="istio-proxy"
Sep 14 11:04:41.164249 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.164228    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.168595 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.168574    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-73290f4b-serving-cert\""
Sep 14 11:04:41.183216 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.183194    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"]
Sep 14 11:04:41.299570 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.299534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz7x2\" (UniqueName: \"kubernetes.io/projected/b7df6120-7c35-44b5-8382-3b883f6bab4d-kube-api-access-jz7x2\") pod \"maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.299744 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.299585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7df6120-7c35-44b5-8382-3b883f6bab4d-openshift-service-ca\") pod \"maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.299744 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.299688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7df6120-7c35-44b5-8382-3b883f6bab4d-maas-api-tls\") pod \"maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.299855 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.299748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7df6120-7c35-44b5-8382-3b883f6bab4d-metrics-certs\") pod \"maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.400959 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.400927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7df6120-7c35-44b5-8382-3b883f6bab4d-maas-api-tls\") pod \"maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.401130 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.400970    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7df6120-7c35-44b5-8382-3b883f6bab4d-metrics-certs\") pod \"maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.401130 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.401094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz7x2\" (UniqueName: \"kubernetes.io/projected/b7df6120-7c35-44b5-8382-3b883f6bab4d-kube-api-access-jz7x2\") pod \"maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.401272 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.401149    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7df6120-7c35-44b5-8382-3b883f6bab4d-openshift-service-ca\") pod \"maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.401790 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.401743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7df6120-7c35-44b5-8382-3b883f6bab4d-openshift-service-ca\") pod \"maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.403338 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.403315    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7df6120-7c35-44b5-8382-3b883f6bab4d-maas-api-tls\") pod \"maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.403447 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.403363    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7df6120-7c35-44b5-8382-3b883f6bab4d-metrics-certs\") pod \"maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.412997 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.412940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz7x2\" (UniqueName: \"kubernetes.io/projected/b7df6120-7c35-44b5-8382-3b883f6bab4d-kube-api-access-jz7x2\") pod \"maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.474605 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.474571    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:41.618722 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.618678    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:04:41.750919 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.750886    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"]
Sep 14 11:04:41.752212 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:04:41.752178    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7df6120_7c35_44b5_8382_3b883f6bab4d.slice/crio-51bb9a46c80057fe6586b87e56f249dbc267c23b17a9990587ba86b331199a6e WatchSource:0}: Error finding container 51bb9a46c80057fe6586b87e56f249dbc267c23b17a9990587ba86b331199a6e: Status 404 returned error can't find the container with id 51bb9a46c80057fe6586b87e56f249dbc267c23b17a9990587ba86b331199a6e
Sep 14 11:04:41.804303 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.804276    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30b6e067-afb7-4c3f-aa4d-44a836aff3ae" path="/var/lib/kubelet/pods/30b6e067-afb7-4c3f-aa4d-44a836aff3ae/volumes"
Sep 14 11:04:41.893083 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:41.893036    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8" event={"ID":"b7df6120-7c35-44b5-8382-3b883f6bab4d","Type":"ContainerStarted","Data":"51bb9a46c80057fe6586b87e56f249dbc267c23b17a9990587ba86b331199a6e"}
Sep 14 11:04:42.900993 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:42.900960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8" event={"ID":"b7df6120-7c35-44b5-8382-3b883f6bab4d","Type":"ContainerStarted","Data":"9226a2d7e09cabd3ad715072aff32694198079e2bba0f6bef9cd3270a8732f60"}
Sep 14 11:04:42.901367 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:42.901083    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:42.920332 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:42.920287    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8" podStartSLOduration=1.486290868 podStartE2EDuration="1.92027199s" totalImagesPullingTime="433.981122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:04:42.919357554 +0000 UTC m=+987.604041944" watchObservedRunningTime="2026-09-14 11:04:42.92027199 +0000 UTC m=+987.604956378"
Sep 14 11:04:48.912101 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:48.912069    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:04:57.925656 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:04:57.925619    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:05:20.058948 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.058913    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"]
Sep 14 11:05:20.059591 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.059164    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8" podUID="b7df6120-7c35-44b5-8382-3b883f6bab4d" containerName="maas-api" containerID="cri-o://9226a2d7e09cabd3ad715072aff32694198079e2bba0f6bef9cd3270a8732f60" gracePeriod=30
Sep 14 11:05:20.319949 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.319885    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:05:20.481670 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.481640    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7df6120-7c35-44b5-8382-3b883f6bab4d-openshift-service-ca\") pod \"b7df6120-7c35-44b5-8382-3b883f6bab4d\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") "
Sep 14 11:05:20.481838 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.481684    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7df6120-7c35-44b5-8382-3b883f6bab4d-maas-api-tls\") pod \"b7df6120-7c35-44b5-8382-3b883f6bab4d\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") "
Sep 14 11:05:20.481838 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.481768    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz7x2\" (UniqueName: \"kubernetes.io/projected/b7df6120-7c35-44b5-8382-3b883f6bab4d-kube-api-access-jz7x2\") pod \"b7df6120-7c35-44b5-8382-3b883f6bab4d\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") "
Sep 14 11:05:20.481954 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.481897    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7df6120-7c35-44b5-8382-3b883f6bab4d-metrics-certs\") pod \"b7df6120-7c35-44b5-8382-3b883f6bab4d\" (UID: \"b7df6120-7c35-44b5-8382-3b883f6bab4d\") "
Sep 14 11:05:20.482043 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.482016    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7df6120-7c35-44b5-8382-3b883f6bab4d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7df6120-7c35-44b5-8382-3b883f6bab4d" (UID: "b7df6120-7c35-44b5-8382-3b883f6bab4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:05:20.482220 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.482194    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7df6120-7c35-44b5-8382-3b883f6bab4d-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:05:20.484114 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.484086    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7df6120-7c35-44b5-8382-3b883f6bab4d-kube-api-access-jz7x2" (OuterVolumeSpecName: "kube-api-access-jz7x2") pod "b7df6120-7c35-44b5-8382-3b883f6bab4d" (UID: "b7df6120-7c35-44b5-8382-3b883f6bab4d"). InnerVolumeSpecName "kube-api-access-jz7x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:05:20.484232 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.484096    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7df6120-7c35-44b5-8382-3b883f6bab4d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b7df6120-7c35-44b5-8382-3b883f6bab4d" (UID: "b7df6120-7c35-44b5-8382-3b883f6bab4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:05:20.484232 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.484111    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7df6120-7c35-44b5-8382-3b883f6bab4d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7df6120-7c35-44b5-8382-3b883f6bab4d" (UID: "b7df6120-7c35-44b5-8382-3b883f6bab4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:05:20.583531 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.583453    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7df6120-7c35-44b5-8382-3b883f6bab4d-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:05:20.583531 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.583481    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7df6120-7c35-44b5-8382-3b883f6bab4d-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:05:20.583531 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:20.583492    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz7x2\" (UniqueName: \"kubernetes.io/projected/b7df6120-7c35-44b5-8382-3b883f6bab4d-kube-api-access-jz7x2\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:05:21.145669 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:21.145636    2561 generic.go:356] "Generic (PLEG): container finished" podID="b7df6120-7c35-44b5-8382-3b883f6bab4d" containerID="9226a2d7e09cabd3ad715072aff32694198079e2bba0f6bef9cd3270a8732f60" exitCode=0
Sep 14 11:05:21.146143 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:21.145679    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"
Sep 14 11:05:21.146143 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:21.145742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8" event={"ID":"b7df6120-7c35-44b5-8382-3b883f6bab4d","Type":"ContainerDied","Data":"9226a2d7e09cabd3ad715072aff32694198079e2bba0f6bef9cd3270a8732f60"}
Sep 14 11:05:21.146143 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:21.145807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8" event={"ID":"b7df6120-7c35-44b5-8382-3b883f6bab4d","Type":"ContainerDied","Data":"51bb9a46c80057fe6586b87e56f249dbc267c23b17a9990587ba86b331199a6e"}
Sep 14 11:05:21.146143 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:21.145842    2561 scope.go:117] "RemoveContainer" containerID="9226a2d7e09cabd3ad715072aff32694198079e2bba0f6bef9cd3270a8732f60"
Sep 14 11:05:21.156078 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:21.156062    2561 scope.go:117] "RemoveContainer" containerID="9226a2d7e09cabd3ad715072aff32694198079e2bba0f6bef9cd3270a8732f60"
Sep 14 11:05:21.156330 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:05:21.156310    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9226a2d7e09cabd3ad715072aff32694198079e2bba0f6bef9cd3270a8732f60\": container with ID starting with 9226a2d7e09cabd3ad715072aff32694198079e2bba0f6bef9cd3270a8732f60 not found: ID does not exist" containerID="9226a2d7e09cabd3ad715072aff32694198079e2bba0f6bef9cd3270a8732f60"
Sep 14 11:05:21.156393 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:21.156339    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9226a2d7e09cabd3ad715072aff32694198079e2bba0f6bef9cd3270a8732f60"} err="failed to get container status \"9226a2d7e09cabd3ad715072aff32694198079e2bba0f6bef9cd3270a8732f60\": rpc error: code = NotFound desc = could not find container \"9226a2d7e09cabd3ad715072aff32694198079e2bba0f6bef9cd3270a8732f60\": container with ID starting with 9226a2d7e09cabd3ad715072aff32694198079e2bba0f6bef9cd3270a8732f60 not found: ID does not exist"
Sep 14 11:05:21.173751 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:21.173729    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"]
Sep 14 11:05:21.179568 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:21.179549    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73290f4b-5cdcc59d79-9vjp8"]
Sep 14 11:05:21.804859 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:21.804828    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7df6120-7c35-44b5-8382-3b883f6bab4d" path="/var/lib/kubelet/pods/b7df6120-7c35-44b5-8382-3b883f6bab4d/volumes"
Sep 14 11:05:26.328921 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:26.328884    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:05:27.805508 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:27.805475    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"]
Sep 14 11:05:27.805945 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:27.805931    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7df6120-7c35-44b5-8382-3b883f6bab4d" containerName="maas-api"
Sep 14 11:05:27.805945 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:27.805947    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7df6120-7c35-44b5-8382-3b883f6bab4d" containerName="maas-api"
Sep 14 11:05:27.806044 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:27.806024    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7df6120-7c35-44b5-8382-3b883f6bab4d" containerName="maas-api"
Sep 14 11:05:27.810630 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:27.810613    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:27.815230 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:27.815210    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1d16dc05-serving-cert\""
Sep 14 11:05:27.825119 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:27.825096    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"]
Sep 14 11:05:27.960399 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:27.960372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/925d51e8-74c6-4421-8b5b-f7de0fff9abc-openshift-service-ca\") pod \"maas-api-e2e-ait-1d16dc05-b985567df-khqwg\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:27.960562 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:27.960412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6f4z\" (UniqueName: \"kubernetes.io/projected/925d51e8-74c6-4421-8b5b-f7de0fff9abc-kube-api-access-j6f4z\") pod \"maas-api-e2e-ait-1d16dc05-b985567df-khqwg\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:27.960562 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:27.960432    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/925d51e8-74c6-4421-8b5b-f7de0fff9abc-metrics-certs\") pod \"maas-api-e2e-ait-1d16dc05-b985567df-khqwg\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:27.960562 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:27.960480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/925d51e8-74c6-4421-8b5b-f7de0fff9abc-maas-api-tls\") pod \"maas-api-e2e-ait-1d16dc05-b985567df-khqwg\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:28.061335 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:28.061245    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/925d51e8-74c6-4421-8b5b-f7de0fff9abc-openshift-service-ca\") pod \"maas-api-e2e-ait-1d16dc05-b985567df-khqwg\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:28.061335 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:28.061294    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6f4z\" (UniqueName: \"kubernetes.io/projected/925d51e8-74c6-4421-8b5b-f7de0fff9abc-kube-api-access-j6f4z\") pod \"maas-api-e2e-ait-1d16dc05-b985567df-khqwg\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:28.061335 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:28.061312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/925d51e8-74c6-4421-8b5b-f7de0fff9abc-metrics-certs\") pod \"maas-api-e2e-ait-1d16dc05-b985567df-khqwg\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:28.061335 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:28.061336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/925d51e8-74c6-4421-8b5b-f7de0fff9abc-maas-api-tls\") pod \"maas-api-e2e-ait-1d16dc05-b985567df-khqwg\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:28.061963 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:28.061934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/925d51e8-74c6-4421-8b5b-f7de0fff9abc-openshift-service-ca\") pod \"maas-api-e2e-ait-1d16dc05-b985567df-khqwg\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:28.063931 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:28.063909    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/925d51e8-74c6-4421-8b5b-f7de0fff9abc-maas-api-tls\") pod \"maas-api-e2e-ait-1d16dc05-b985567df-khqwg\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:28.064033 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:28.063938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/925d51e8-74c6-4421-8b5b-f7de0fff9abc-metrics-certs\") pod \"maas-api-e2e-ait-1d16dc05-b985567df-khqwg\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:28.069596 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:28.069565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6f4z\" (UniqueName: \"kubernetes.io/projected/925d51e8-74c6-4421-8b5b-f7de0fff9abc-kube-api-access-j6f4z\") pod \"maas-api-e2e-ait-1d16dc05-b985567df-khqwg\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:28.120498 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:28.120468    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:28.401100 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:28.401062    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"]
Sep 14 11:05:28.402117 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:05:28.402085    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925d51e8_74c6_4421_8b5b_f7de0fff9abc.slice/crio-32c4e3ec21273b187d28cbba6ddeb17f28e51785986391a102fa17ae097ffbe0 WatchSource:0}: Error finding container 32c4e3ec21273b187d28cbba6ddeb17f28e51785986391a102fa17ae097ffbe0: Status 404 returned error can't find the container with id 32c4e3ec21273b187d28cbba6ddeb17f28e51785986391a102fa17ae097ffbe0
Sep 14 11:05:29.208366 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:29.208333    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg" event={"ID":"925d51e8-74c6-4421-8b5b-f7de0fff9abc","Type":"ContainerStarted","Data":"32c4e3ec21273b187d28cbba6ddeb17f28e51785986391a102fa17ae097ffbe0"}
Sep 14 11:05:31.223179 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:31.223147    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg" event={"ID":"925d51e8-74c6-4421-8b5b-f7de0fff9abc","Type":"ContainerStarted","Data":"b5a5a09220a9d573b399a591674650cbd5b08e3931935088b7fc96ad2befb987"}
Sep 14 11:05:31.223542 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:31.223334    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:31.242486 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:31.242441    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg" podStartSLOduration=2.212056734 podStartE2EDuration="4.242426106s" totalImagesPullingTime="2.030369372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:05:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:05:31.241238355 +0000 UTC m=+1035.925922744" watchObservedRunningTime="2026-09-14 11:05:31.242426106 +0000 UTC m=+1035.927110567"
Sep 14 11:05:33.744628 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.744595    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"]
Sep 14 11:05:33.749021 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.749000    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.753321 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.753293    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b275d0\"/\"autores-model-b275d0-kserve-self-signed-certs\""
Sep 14 11:05:33.753321 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.753317    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b275d0\"/\"openshift-service-ca.crt\""
Sep 14 11:05:33.753489 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.753462    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b275d0\"/\"default-dockercfg-drgb4\""
Sep 14 11:05:33.753576 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.753559    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b275d0\"/\"kube-root-ca.crt\""
Sep 14 11:05:33.768807 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.768785    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"]
Sep 14 11:05:33.817900 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.817875    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-model-cache\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.818015 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.817913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af70f4d7-f00f-4c1b-9b35-30a61a48df72-tls-certs\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.818015 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.817937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-home\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.818015 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.817996    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-dshm\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.818123 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.818038    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ps6d\" (UniqueName: \"kubernetes.io/projected/af70f4d7-f00f-4c1b-9b35-30a61a48df72-kube-api-access-6ps6d\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.818182 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.818164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-tmp-dir\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.919108 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.919081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-home\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.919264 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.919116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-dshm\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.919264 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.919139    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ps6d\" (UniqueName: \"kubernetes.io/projected/af70f4d7-f00f-4c1b-9b35-30a61a48df72-kube-api-access-6ps6d\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.919264 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.919205    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-tmp-dir\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.919429 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.919408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-model-cache\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.919490 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.919479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af70f4d7-f00f-4c1b-9b35-30a61a48df72-tls-certs\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.919547 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.919502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-home\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.919597 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.919571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-tmp-dir\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.919734 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.919695    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-model-cache\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.921449 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.921422    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-dshm\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.921817 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.921794    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af70f4d7-f00f-4c1b-9b35-30a61a48df72-tls-certs\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:33.934480 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:33.934458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ps6d\" (UniqueName: \"kubernetes.io/projected/af70f4d7-f00f-4c1b-9b35-30a61a48df72-kube-api-access-6ps6d\") pod \"autores-model-b275d0-kserve-6bf8b59499-5nj9g\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") " pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:34.058916 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:34.058830    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:34.347390 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:34.347362    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"]
Sep 14 11:05:34.348619 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:05:34.348589    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf70f4d7_f00f_4c1b_9b35_30a61a48df72.slice/crio-c94b858e28ce7b893df802d2c3e369b2ea47a4c42a3ddf0408f943ee1be307fa WatchSource:0}: Error finding container c94b858e28ce7b893df802d2c3e369b2ea47a4c42a3ddf0408f943ee1be307fa: Status 404 returned error can't find the container with id c94b858e28ce7b893df802d2c3e369b2ea47a4c42a3ddf0408f943ee1be307fa
Sep 14 11:05:35.248941 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:35.248895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g" event={"ID":"af70f4d7-f00f-4c1b-9b35-30a61a48df72","Type":"ContainerStarted","Data":"2f6130c2dcdf0eb0542b67f24a74c8aad4c966837f3b46ff3ea487abd69cb4a1"}
Sep 14 11:05:35.249317 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:35.248953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g" event={"ID":"af70f4d7-f00f-4c1b-9b35-30a61a48df72","Type":"ContainerStarted","Data":"c94b858e28ce7b893df802d2c3e369b2ea47a4c42a3ddf0408f943ee1be307fa"}
Sep 14 11:05:37.234492 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:37.234463    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:05:37.258119 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:37.258061    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g" podStartSLOduration=4.258044 podStartE2EDuration="4.258044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:05:35.269059918 +0000 UTC m=+1039.953744306" watchObservedRunningTime="2026-09-14 11:05:37.258044 +0000 UTC m=+1041.942728390"
Sep 14 11:05:44.059120 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:44.059086    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:44.059662 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:44.059138    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:44.072372 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:44.072347    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:44.317527 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:44.317445    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:05:50.098840 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:05:50.098810    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:06:06.954059 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:06.954022    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"]
Sep 14 11:06:06.954636 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:06.954252    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg" podUID="925d51e8-74c6-4421-8b5b-f7de0fff9abc" containerName="maas-api" containerID="cri-o://b5a5a09220a9d573b399a591674650cbd5b08e3931935088b7fc96ad2befb987" gracePeriod=30
Sep 14 11:06:07.205252 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.205196    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:06:07.253372 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.253339    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/925d51e8-74c6-4421-8b5b-f7de0fff9abc-metrics-certs\") pod \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") "
Sep 14 11:06:07.253570 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.253437    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6f4z\" (UniqueName: \"kubernetes.io/projected/925d51e8-74c6-4421-8b5b-f7de0fff9abc-kube-api-access-j6f4z\") pod \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") "
Sep 14 11:06:07.253570 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.253464    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/925d51e8-74c6-4421-8b5b-f7de0fff9abc-maas-api-tls\") pod \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") "
Sep 14 11:06:07.253695 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.253599    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/925d51e8-74c6-4421-8b5b-f7de0fff9abc-openshift-service-ca\") pod \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\" (UID: \"925d51e8-74c6-4421-8b5b-f7de0fff9abc\") "
Sep 14 11:06:07.254269 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.254228    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925d51e8-74c6-4421-8b5b-f7de0fff9abc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "925d51e8-74c6-4421-8b5b-f7de0fff9abc" (UID: "925d51e8-74c6-4421-8b5b-f7de0fff9abc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:06:07.255954 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.255931    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925d51e8-74c6-4421-8b5b-f7de0fff9abc-kube-api-access-j6f4z" (OuterVolumeSpecName: "kube-api-access-j6f4z") pod "925d51e8-74c6-4421-8b5b-f7de0fff9abc" (UID: "925d51e8-74c6-4421-8b5b-f7de0fff9abc"). InnerVolumeSpecName "kube-api-access-j6f4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:06:07.256116 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.256077    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925d51e8-74c6-4421-8b5b-f7de0fff9abc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "925d51e8-74c6-4421-8b5b-f7de0fff9abc" (UID: "925d51e8-74c6-4421-8b5b-f7de0fff9abc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:06:07.256116 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.256094    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925d51e8-74c6-4421-8b5b-f7de0fff9abc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "925d51e8-74c6-4421-8b5b-f7de0fff9abc" (UID: "925d51e8-74c6-4421-8b5b-f7de0fff9abc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:06:07.355510 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.355458    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/925d51e8-74c6-4421-8b5b-f7de0fff9abc-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:07.355510 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.355493    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/925d51e8-74c6-4421-8b5b-f7de0fff9abc-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:07.355510 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.355515    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6f4z\" (UniqueName: \"kubernetes.io/projected/925d51e8-74c6-4421-8b5b-f7de0fff9abc-kube-api-access-j6f4z\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:07.355821 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.355531    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/925d51e8-74c6-4421-8b5b-f7de0fff9abc-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:07.457304 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.457214    2561 generic.go:356] "Generic (PLEG): container finished" podID="925d51e8-74c6-4421-8b5b-f7de0fff9abc" containerID="b5a5a09220a9d573b399a591674650cbd5b08e3931935088b7fc96ad2befb987" exitCode=0
Sep 14 11:06:07.457304 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.457262    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"
Sep 14 11:06:07.457527 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.457303    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg" event={"ID":"925d51e8-74c6-4421-8b5b-f7de0fff9abc","Type":"ContainerDied","Data":"b5a5a09220a9d573b399a591674650cbd5b08e3931935088b7fc96ad2befb987"}
Sep 14 11:06:07.457527 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.457371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg" event={"ID":"925d51e8-74c6-4421-8b5b-f7de0fff9abc","Type":"ContainerDied","Data":"32c4e3ec21273b187d28cbba6ddeb17f28e51785986391a102fa17ae097ffbe0"}
Sep 14 11:06:07.457527 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.457403    2561 scope.go:117] "RemoveContainer" containerID="b5a5a09220a9d573b399a591674650cbd5b08e3931935088b7fc96ad2befb987"
Sep 14 11:06:07.468246 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.468230    2561 scope.go:117] "RemoveContainer" containerID="b5a5a09220a9d573b399a591674650cbd5b08e3931935088b7fc96ad2befb987"
Sep 14 11:06:07.468514 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:06:07.468494    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a5a09220a9d573b399a591674650cbd5b08e3931935088b7fc96ad2befb987\": container with ID starting with b5a5a09220a9d573b399a591674650cbd5b08e3931935088b7fc96ad2befb987 not found: ID does not exist" containerID="b5a5a09220a9d573b399a591674650cbd5b08e3931935088b7fc96ad2befb987"
Sep 14 11:06:07.468593 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.468525    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a5a09220a9d573b399a591674650cbd5b08e3931935088b7fc96ad2befb987"} err="failed to get container status \"b5a5a09220a9d573b399a591674650cbd5b08e3931935088b7fc96ad2befb987\": rpc error: code = NotFound desc = could not find container \"b5a5a09220a9d573b399a591674650cbd5b08e3931935088b7fc96ad2befb987\": container with ID starting with b5a5a09220a9d573b399a591674650cbd5b08e3931935088b7fc96ad2befb987 not found: ID does not exist"
Sep 14 11:06:07.482015 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.481994    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"]
Sep 14 11:06:07.484531 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.484506    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d16dc05-b985567df-khqwg"]
Sep 14 11:06:07.804208 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:07.804112    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="925d51e8-74c6-4421-8b5b-f7de0fff9abc" path="/var/lib/kubelet/pods/925d51e8-74c6-4421-8b5b-f7de0fff9abc/volumes"
Sep 14 11:06:12.362672 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:12.362632    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:06:18.079173 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.079138    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"]
Sep 14 11:06:18.079806 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.079790    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="925d51e8-74c6-4421-8b5b-f7de0fff9abc" containerName="maas-api"
Sep 14 11:06:18.079862 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.079809    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="925d51e8-74c6-4421-8b5b-f7de0fff9abc" containerName="maas-api"
Sep 14 11:06:18.079920 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.079908    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="925d51e8-74c6-4421-8b5b-f7de0fff9abc" containerName="maas-api"
Sep 14 11:06:18.084754 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.084734    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:18.088324 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.088307    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-bcc15f82-serving-cert\""
Sep 14 11:06:18.115494 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.115466    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"]
Sep 14 11:06:18.157128 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.157096    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6fvs\" (UniqueName: \"kubernetes.io/projected/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-kube-api-access-r6fvs\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:18.157248 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.157234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-metrics-certs\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:18.157295 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.157257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-openshift-service-ca\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:18.157295 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.157275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-maas-api-tls\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:18.258355 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.258319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-metrics-certs\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:18.258524 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.258367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-openshift-service-ca\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:18.258524 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.258420    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-maas-api-tls\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:18.258524 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.258477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6fvs\" (UniqueName: \"kubernetes.io/projected/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-kube-api-access-r6fvs\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:18.258698 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:06:18.258601    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-serving-cert: secret "maas-api-e2e-derive-bcc15f82-serving-cert" not found
Sep 14 11:06:18.258698 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:06:18.258670    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-maas-api-tls podName:e3672651-6ab0-4403-a19e-c0f6aedfcc8e nodeName:}" failed. No retries permitted until 2026-09-14 11:06:18.758650816 +0000 UTC m=+1083.443335195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-maas-api-tls") pod "maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j" (UID: "e3672651-6ab0-4403-a19e-c0f6aedfcc8e") : secret "maas-api-e2e-derive-bcc15f82-serving-cert" not found
Sep 14 11:06:18.259159 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.259139    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-openshift-service-ca\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:18.260928 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.260905    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-metrics-certs\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:18.279751 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.279727    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6fvs\" (UniqueName: \"kubernetes.io/projected/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-kube-api-access-r6fvs\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:18.763940 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:18.763905    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-maas-api-tls\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:18.764126 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:06:18.764033    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-serving-cert: secret "maas-api-e2e-derive-bcc15f82-serving-cert" not found
Sep 14 11:06:18.764126 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:06:18.764101    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-maas-api-tls podName:e3672651-6ab0-4403-a19e-c0f6aedfcc8e nodeName:}" failed. No retries permitted until 2026-09-14 11:06:19.764082749 +0000 UTC m=+1084.448767123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-maas-api-tls") pod "maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j" (UID: "e3672651-6ab0-4403-a19e-c0f6aedfcc8e") : secret "maas-api-e2e-derive-bcc15f82-serving-cert" not found
Sep 14 11:06:19.773552 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:19.773516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-maas-api-tls\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:19.775921 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:19.775900    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-maas-api-tls\") pod \"maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:19.895596 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:19.895568    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:20.192962 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:20.192930    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"]
Sep 14 11:06:20.194195 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:06:20.194169    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3672651_6ab0_4403_a19e_c0f6aedfcc8e.slice/crio-c34d1c7fee3623aaa2848e401cb157901d9814fab35862642bbde4d8e430fa5b WatchSource:0}: Error finding container c34d1c7fee3623aaa2848e401cb157901d9814fab35862642bbde4d8e430fa5b: Status 404 returned error can't find the container with id c34d1c7fee3623aaa2848e401cb157901d9814fab35862642bbde4d8e430fa5b
Sep 14 11:06:20.537223 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:20.537141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j" event={"ID":"e3672651-6ab0-4403-a19e-c0f6aedfcc8e","Type":"ContainerStarted","Data":"c34d1c7fee3623aaa2848e401cb157901d9814fab35862642bbde4d8e430fa5b"}
Sep 14 11:06:22.558972 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:22.558928    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j" event={"ID":"e3672651-6ab0-4403-a19e-c0f6aedfcc8e","Type":"ContainerStarted","Data":"1ec5000d732de6c0e307dd788352e6d74819c215a1af34b76e21baf67356efa8"}
Sep 14 11:06:22.559374 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:22.559042    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:22.586787 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:22.586740    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j" podStartSLOduration=2.816126072 podStartE2EDuration="4.586725719s" totalImagesPullingTime="1.770599647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:22.583706861 +0000 UTC m=+1087.268391254" watchObservedRunningTime="2026-09-14 11:06:22.586725719 +0000 UTC m=+1087.271410105"
Sep 14 11:06:26.081740 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:26.081690    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"]
Sep 14 11:06:28.569854 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:28.569821    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:37.424855 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:37.424819    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:06:41.102696 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.102647    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g" podUID="af70f4d7-f00f-4c1b-9b35-30a61a48df72" containerName="main" containerID="cri-o://2f6130c2dcdf0eb0542b67f24a74c8aad4c966837f3b46ff3ea487abd69cb4a1" gracePeriod=45
Sep 14 11:06:41.357925 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.357903    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:06:41.487024 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.486991    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-model-cache\") pod \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") "
Sep 14 11:06:41.487187 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.487070    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-tmp-dir\") pod \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") "
Sep 14 11:06:41.487187 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.487157    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-dshm\") pod \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") "
Sep 14 11:06:41.487272 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.487196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-home\") pod \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") "
Sep 14 11:06:41.487272 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.487235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af70f4d7-f00f-4c1b-9b35-30a61a48df72-tls-certs\") pod \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") "
Sep 14 11:06:41.487272 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.487265    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ps6d\" (UniqueName: \"kubernetes.io/projected/af70f4d7-f00f-4c1b-9b35-30a61a48df72-kube-api-access-6ps6d\") pod \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\" (UID: \"af70f4d7-f00f-4c1b-9b35-30a61a48df72\") "
Sep 14 11:06:41.487421 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.487298    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-model-cache" (OuterVolumeSpecName: "model-cache") pod "af70f4d7-f00f-4c1b-9b35-30a61a48df72" (UID: "af70f4d7-f00f-4c1b-9b35-30a61a48df72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:06:41.487421 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.487361    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af70f4d7-f00f-4c1b-9b35-30a61a48df72" (UID: "af70f4d7-f00f-4c1b-9b35-30a61a48df72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:06:41.487534 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.487445    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-home" (OuterVolumeSpecName: "home") pod "af70f4d7-f00f-4c1b-9b35-30a61a48df72" (UID: "af70f4d7-f00f-4c1b-9b35-30a61a48df72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:06:41.487668 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.487650    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-model-cache\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:41.487771 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.487670    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-tmp-dir\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:41.487771 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.487679    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-home\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:41.489362 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.489336    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-dshm" (OuterVolumeSpecName: "dshm") pod "af70f4d7-f00f-4c1b-9b35-30a61a48df72" (UID: "af70f4d7-f00f-4c1b-9b35-30a61a48df72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:06:41.489472 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.489372    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af70f4d7-f00f-4c1b-9b35-30a61a48df72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af70f4d7-f00f-4c1b-9b35-30a61a48df72" (UID: "af70f4d7-f00f-4c1b-9b35-30a61a48df72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:06:41.489472 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.489398    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af70f4d7-f00f-4c1b-9b35-30a61a48df72-kube-api-access-6ps6d" (OuterVolumeSpecName: "kube-api-access-6ps6d") pod "af70f4d7-f00f-4c1b-9b35-30a61a48df72" (UID: "af70f4d7-f00f-4c1b-9b35-30a61a48df72"). InnerVolumeSpecName "kube-api-access-6ps6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:06:41.589155 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.589129    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af70f4d7-f00f-4c1b-9b35-30a61a48df72-dshm\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:41.589155 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.589150    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af70f4d7-f00f-4c1b-9b35-30a61a48df72-tls-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:41.589306 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.589161    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ps6d\" (UniqueName: \"kubernetes.io/projected/af70f4d7-f00f-4c1b-9b35-30a61a48df72-kube-api-access-6ps6d\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:41.694818 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.694741    2561 generic.go:356] "Generic (PLEG): container finished" podID="af70f4d7-f00f-4c1b-9b35-30a61a48df72" containerID="2f6130c2dcdf0eb0542b67f24a74c8aad4c966837f3b46ff3ea487abd69cb4a1" exitCode=0
Sep 14 11:06:41.694818 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.694781    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"
Sep 14 11:06:41.694980 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.694819    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g" event={"ID":"af70f4d7-f00f-4c1b-9b35-30a61a48df72","Type":"ContainerDied","Data":"2f6130c2dcdf0eb0542b67f24a74c8aad4c966837f3b46ff3ea487abd69cb4a1"}
Sep 14 11:06:41.694980 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.694860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g" event={"ID":"af70f4d7-f00f-4c1b-9b35-30a61a48df72","Type":"ContainerDied","Data":"c94b858e28ce7b893df802d2c3e369b2ea47a4c42a3ddf0408f943ee1be307fa"}
Sep 14 11:06:41.694980 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.694877    2561 scope.go:117] "RemoveContainer" containerID="2f6130c2dcdf0eb0542b67f24a74c8aad4c966837f3b46ff3ea487abd69cb4a1"
Sep 14 11:06:41.704634 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.704603    2561 scope.go:117] "RemoveContainer" containerID="2f6130c2dcdf0eb0542b67f24a74c8aad4c966837f3b46ff3ea487abd69cb4a1"
Sep 14 11:06:41.704903 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:06:41.704883    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6130c2dcdf0eb0542b67f24a74c8aad4c966837f3b46ff3ea487abd69cb4a1\": container with ID starting with 2f6130c2dcdf0eb0542b67f24a74c8aad4c966837f3b46ff3ea487abd69cb4a1 not found: ID does not exist" containerID="2f6130c2dcdf0eb0542b67f24a74c8aad4c966837f3b46ff3ea487abd69cb4a1"
Sep 14 11:06:41.704980 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.704910    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6130c2dcdf0eb0542b67f24a74c8aad4c966837f3b46ff3ea487abd69cb4a1"} err="failed to get container status \"2f6130c2dcdf0eb0542b67f24a74c8aad4c966837f3b46ff3ea487abd69cb4a1\": rpc error: code = NotFound desc = could not find container \"2f6130c2dcdf0eb0542b67f24a74c8aad4c966837f3b46ff3ea487abd69cb4a1\": container with ID starting with 2f6130c2dcdf0eb0542b67f24a74c8aad4c966837f3b46ff3ea487abd69cb4a1 not found: ID does not exist"
Sep 14 11:06:41.731909 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.731885    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"]
Sep 14 11:06:41.737865 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.737845    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-b275d0/autores-model-b275d0-kserve-6bf8b59499-5nj9g"]
Sep 14 11:06:41.803906 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:41.803882    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af70f4d7-f00f-4c1b-9b35-30a61a48df72" path="/var/lib/kubelet/pods/af70f4d7-f00f-4c1b-9b35-30a61a48df72/volumes"
Sep 14 11:06:42.265884 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.265854    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"]
Sep 14 11:06:42.266284 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.266064    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j" podUID="e3672651-6ab0-4403-a19e-c0f6aedfcc8e" containerName="maas-api" containerID="cri-o://1ec5000d732de6c0e307dd788352e6d74819c215a1af34b76e21baf67356efa8" gracePeriod=30
Sep 14 11:06:42.528474 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.528412    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:42.701016 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.700157    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-openshift-service-ca\") pod \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") "
Sep 14 11:06:42.701016 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.700236    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-metrics-certs\") pod \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") "
Sep 14 11:06:42.701016 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.700315    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-maas-api-tls\") pod \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") "
Sep 14 11:06:42.701016 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.700396    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6fvs\" (UniqueName: \"kubernetes.io/projected/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-kube-api-access-r6fvs\") pod \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\" (UID: \"e3672651-6ab0-4403-a19e-c0f6aedfcc8e\") "
Sep 14 11:06:42.701016 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.700942    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3672651-6ab0-4403-a19e-c0f6aedfcc8e" (UID: "e3672651-6ab0-4403-a19e-c0f6aedfcc8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:06:42.703663 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.703620    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3672651-6ab0-4403-a19e-c0f6aedfcc8e" (UID: "e3672651-6ab0-4403-a19e-c0f6aedfcc8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:06:42.705626 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.705560    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-kube-api-access-r6fvs" (OuterVolumeSpecName: "kube-api-access-r6fvs") pod "e3672651-6ab0-4403-a19e-c0f6aedfcc8e" (UID: "e3672651-6ab0-4403-a19e-c0f6aedfcc8e"). InnerVolumeSpecName "kube-api-access-r6fvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:06:42.706912 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.706871    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3672651-6ab0-4403-a19e-c0f6aedfcc8e" (UID: "e3672651-6ab0-4403-a19e-c0f6aedfcc8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:06:42.719387 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.719360    2561 generic.go:356] "Generic (PLEG): container finished" podID="e3672651-6ab0-4403-a19e-c0f6aedfcc8e" containerID="1ec5000d732de6c0e307dd788352e6d74819c215a1af34b76e21baf67356efa8" exitCode=0
Sep 14 11:06:42.719496 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.719405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j" event={"ID":"e3672651-6ab0-4403-a19e-c0f6aedfcc8e","Type":"ContainerDied","Data":"1ec5000d732de6c0e307dd788352e6d74819c215a1af34b76e21baf67356efa8"}
Sep 14 11:06:42.719496 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.719414    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"
Sep 14 11:06:42.719496 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.719457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j" event={"ID":"e3672651-6ab0-4403-a19e-c0f6aedfcc8e","Type":"ContainerDied","Data":"c34d1c7fee3623aaa2848e401cb157901d9814fab35862642bbde4d8e430fa5b"}
Sep 14 11:06:42.719496 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.719487    2561 scope.go:117] "RemoveContainer" containerID="1ec5000d732de6c0e307dd788352e6d74819c215a1af34b76e21baf67356efa8"
Sep 14 11:06:42.733913 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.733898    2561 scope.go:117] "RemoveContainer" containerID="1ec5000d732de6c0e307dd788352e6d74819c215a1af34b76e21baf67356efa8"
Sep 14 11:06:42.734179 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:06:42.734160    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec5000d732de6c0e307dd788352e6d74819c215a1af34b76e21baf67356efa8\": container with ID starting with 1ec5000d732de6c0e307dd788352e6d74819c215a1af34b76e21baf67356efa8 not found: ID does not exist" containerID="1ec5000d732de6c0e307dd788352e6d74819c215a1af34b76e21baf67356efa8"
Sep 14 11:06:42.734224 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.734189    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec5000d732de6c0e307dd788352e6d74819c215a1af34b76e21baf67356efa8"} err="failed to get container status \"1ec5000d732de6c0e307dd788352e6d74819c215a1af34b76e21baf67356efa8\": rpc error: code = NotFound desc = could not find container \"1ec5000d732de6c0e307dd788352e6d74819c215a1af34b76e21baf67356efa8\": container with ID starting with 1ec5000d732de6c0e307dd788352e6d74819c215a1af34b76e21baf67356efa8 not found: ID does not exist"
Sep 14 11:06:42.764063 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.764041    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"]
Sep 14 11:06:42.766563 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.766536    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bcc15f82-54b6cb4bd6-swg5j"]
Sep 14 11:06:42.801251 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.801187    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:42.801251 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.801212    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6fvs\" (UniqueName: \"kubernetes.io/projected/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-kube-api-access-r6fvs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:42.801251 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.801228    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:42.801251 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:42.801241    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3672651-6ab0-4403-a19e-c0f6aedfcc8e-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:06:43.803068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:43.803037    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3672651-6ab0-4403-a19e-c0f6aedfcc8e" path="/var/lib/kubelet/pods/e3672651-6ab0-4403-a19e-c0f6aedfcc8e/volumes"
Sep 14 11:06:52.516598 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:06:52.516569    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:07:03.743787 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:03.743749    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"]
Sep 14 11:07:03.744314 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:03.744005    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy" containerID="cri-o://f018ea5dd4928fc5df2fcba11bd67d86186fce42d9cbcd111795fcd6156bf2a8" gracePeriod=30
Sep 14 11:07:08.293881 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:08.293844    2561 patch_prober.go:28] interesting pod/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 11:07:08.294236 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:08.293903    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 11:07:08.902104 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:08.902070    2561 generic.go:356] "Generic (PLEG): container finished" podID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerID="f018ea5dd4928fc5df2fcba11bd67d86186fce42d9cbcd111795fcd6156bf2a8" exitCode=0
Sep 14 11:07:08.902273 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:08.902142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" event={"ID":"7cbc1301-3425-4ef9-952e-9347fbfae04d","Type":"ContainerDied","Data":"f018ea5dd4928fc5df2fcba11bd67d86186fce42d9cbcd111795fcd6156bf2a8"}
Sep 14 11:07:08.902273 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:08.902231    2561 scope.go:117] "RemoveContainer" containerID="7c879c0b9f26a264466f9cb2cc0eea5e76714c757d45446180ede0ca7a0738aa"
Sep 14 11:07:09.009916 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.009857    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:07:09.157443 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.157415    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-envoy\") pod \"7cbc1301-3425-4ef9-952e-9347fbfae04d\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") "
Sep 14 11:07:09.157598 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.157461    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-podinfo\") pod \"7cbc1301-3425-4ef9-952e-9347fbfae04d\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") "
Sep 14 11:07:09.157598 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.157498    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-data\") pod \"7cbc1301-3425-4ef9-952e-9347fbfae04d\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") "
Sep 14 11:07:09.157598 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.157550    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbc1301-3425-4ef9-952e-9347fbfae04d-istiod-ca-cert\") pod \"7cbc1301-3425-4ef9-952e-9347fbfae04d\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") "
Sep 14 11:07:09.157749 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.157614    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmzgm\" (UniqueName: \"kubernetes.io/projected/7cbc1301-3425-4ef9-952e-9347fbfae04d-kube-api-access-rmzgm\") pod \"7cbc1301-3425-4ef9-952e-9347fbfae04d\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") "
Sep 14 11:07:09.157749 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.157640    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-token\") pod \"7cbc1301-3425-4ef9-952e-9347fbfae04d\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") "
Sep 14 11:07:09.157749 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.157695    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-credential-socket\") pod \"7cbc1301-3425-4ef9-952e-9347fbfae04d\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") "
Sep 14 11:07:09.157915 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.157783    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-workload-certs\") pod \"7cbc1301-3425-4ef9-952e-9347fbfae04d\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") "
Sep 14 11:07:09.157915 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.157822    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-workload-socket\") pod \"7cbc1301-3425-4ef9-952e-9347fbfae04d\" (UID: \"7cbc1301-3425-4ef9-952e-9347fbfae04d\") "
Sep 14 11:07:09.157915 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.157861    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-data" (OuterVolumeSpecName: "istio-data") pod "7cbc1301-3425-4ef9-952e-9347fbfae04d" (UID: "7cbc1301-3425-4ef9-952e-9347fbfae04d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:07:09.158094 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.158020    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7cbc1301-3425-4ef9-952e-9347fbfae04d" (UID: "7cbc1301-3425-4ef9-952e-9347fbfae04d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:07:09.158094 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.158076    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7cbc1301-3425-4ef9-952e-9347fbfae04d" (UID: "7cbc1301-3425-4ef9-952e-9347fbfae04d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:07:09.158208 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.158188    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7cbc1301-3425-4ef9-952e-9347fbfae04d" (UID: "7cbc1301-3425-4ef9-952e-9347fbfae04d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:07:09.158267 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.158239    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-credential-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:07:09.158267 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.158258    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-workload-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:07:09.158376 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.158272    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-data\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:07:09.158376 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.158321    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cbc1301-3425-4ef9-952e-9347fbfae04d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7cbc1301-3425-4ef9-952e-9347fbfae04d" (UID: "7cbc1301-3425-4ef9-952e-9347fbfae04d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:07:09.159774 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.159741    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7cbc1301-3425-4ef9-952e-9347fbfae04d" (UID: "7cbc1301-3425-4ef9-952e-9347fbfae04d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:07:09.159895 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.159789    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7cbc1301-3425-4ef9-952e-9347fbfae04d" (UID: "7cbc1301-3425-4ef9-952e-9347fbfae04d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 11:07:09.159895 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.159865    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbc1301-3425-4ef9-952e-9347fbfae04d-kube-api-access-rmzgm" (OuterVolumeSpecName: "kube-api-access-rmzgm") pod "7cbc1301-3425-4ef9-952e-9347fbfae04d" (UID: "7cbc1301-3425-4ef9-952e-9347fbfae04d"). InnerVolumeSpecName "kube-api-access-rmzgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:07:09.160017 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.159948    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-token" (OuterVolumeSpecName: "istio-token") pod "7cbc1301-3425-4ef9-952e-9347fbfae04d" (UID: "7cbc1301-3425-4ef9-952e-9347fbfae04d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:07:09.259351 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.259313    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmzgm\" (UniqueName: \"kubernetes.io/projected/7cbc1301-3425-4ef9-952e-9347fbfae04d-kube-api-access-rmzgm\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:07:09.259351 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.259349    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:07:09.259351 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.259359    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-workload-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:07:09.259551 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.259367    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-envoy\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:07:09.259551 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.259375    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbc1301-3425-4ef9-952e-9347fbfae04d-istio-podinfo\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:07:09.259551 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.259383    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbc1301-3425-4ef9-952e-9347fbfae04d-istiod-ca-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:07:09.533954 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.533914    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:07:09.909111 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.909083    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"
Sep 14 11:07:09.909294 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.909116    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv" event={"ID":"7cbc1301-3425-4ef9-952e-9347fbfae04d","Type":"ContainerDied","Data":"e67fc916a7108b898bee74617e7a409a57172dcb2209a83d0a3d6659f974cd2d"}
Sep 14 11:07:09.909294 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.909172    2561 scope.go:117] "RemoveContainer" containerID="f018ea5dd4928fc5df2fcba11bd67d86186fce42d9cbcd111795fcd6156bf2a8"
Sep 14 11:07:09.929501 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.929476    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"]
Sep 14 11:07:09.938003 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:09.937979    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-b275d0-openshift-default-f5cf4fffd-86gzv"]
Sep 14 11:07:11.459140 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.459106    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"]
Sep 14 11:07:11.459750 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.459730    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3672651-6ab0-4403-a19e-c0f6aedfcc8e" containerName="maas-api"
Sep 14 11:07:11.459750 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.459750    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3672651-6ab0-4403-a19e-c0f6aedfcc8e" containerName="maas-api"
Sep 14 11:07:11.459906 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.459766    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af70f4d7-f00f-4c1b-9b35-30a61a48df72" containerName="main"
Sep 14 11:07:11.459906 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.459774    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="af70f4d7-f00f-4c1b-9b35-30a61a48df72" containerName="main"
Sep 14 11:07:11.459906 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.459786    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy"
Sep 14 11:07:11.459906 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.459794    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy"
Sep 14 11:07:11.459906 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.459813    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy"
Sep 14 11:07:11.459906 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.459821    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy"
Sep 14 11:07:11.460245 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.459986    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3672651-6ab0-4403-a19e-c0f6aedfcc8e" containerName="maas-api"
Sep 14 11:07:11.460245 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.460003    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy"
Sep 14 11:07:11.460245 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.460012    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="af70f4d7-f00f-4c1b-9b35-30a61a48df72" containerName="main"
Sep 14 11:07:11.464900 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.464878    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:11.468285 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.468266    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-cc14fe-serving-cert\""
Sep 14 11:07:11.482097 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.482076    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"]
Sep 14 11:07:11.575376 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.575344    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"]
Sep 14 11:07:11.575872 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.575856    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" containerName="istio-proxy"
Sep 14 11:07:11.578955 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.578939    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.579627 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.579601    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-metrics-certs\") pod \"maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:11.579748 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.579651    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-maas-api-tls\") pod \"maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:11.579810 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.579762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlwsz\" (UniqueName: \"kubernetes.io/projected/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-kube-api-access-rlwsz\") pod \"maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:11.579864 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.579835    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-openshift-service-ca\") pod \"maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:11.583694 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.583673    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0777c1\"/\"openshift-service-ca.crt\""
Sep 14 11:07:11.583800 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.583693    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0777c1\"/\"test-model-0777c1-kserve-self-signed-certs\""
Sep 14 11:07:11.586143 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.585875    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0777c1\"/\"default-dockercfg-qpdfl\""
Sep 14 11:07:11.587394 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.587374    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0777c1\"/\"kube-root-ca.crt\""
Sep 14 11:07:11.608760 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.608740    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"]
Sep 14 11:07:11.681155 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.681131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-dshm\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.681295 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.681192    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-metrics-certs\") pod \"maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:11.681295 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.681213    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmfcc\" (UniqueName: \"kubernetes.io/projected/d4dd768c-253a-463f-bfb5-844bc5c5f243-kube-api-access-mmfcc\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.681295 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.681229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-model-cache\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.681295 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.681255    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-maas-api-tls\") pod \"maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:11.681295 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.681269    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-home\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.681562 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.681353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlwsz\" (UniqueName: \"kubernetes.io/projected/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-kube-api-access-rlwsz\") pod \"maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:11.681562 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:07:11.681382    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-serving-cert: secret "maas-api-e2e-explicit-cc14fe-serving-cert" not found
Sep 14 11:07:11.681562 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.681402    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-tmp-dir\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.681562 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:07:11.681458    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-maas-api-tls podName:bc02c3e9-15eb-4b37-af4c-ad2730e44e0e nodeName:}" failed. No retries permitted until 2026-09-14 11:07:12.181434704 +0000 UTC m=+1136.866119085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-maas-api-tls") pod "maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9" (UID: "bc02c3e9-15eb-4b37-af4c-ad2730e44e0e") : secret "maas-api-e2e-explicit-cc14fe-serving-cert" not found
Sep 14 11:07:11.681562 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.681558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-openshift-service-ca\") pod \"maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:11.681822 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.681590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4dd768c-253a-463f-bfb5-844bc5c5f243-tls-certs\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.682136 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.682120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-openshift-service-ca\") pod \"maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:11.683530 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.683503    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-metrics-certs\") pod \"maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:11.692681 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.692663    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlwsz\" (UniqueName: \"kubernetes.io/projected/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-kube-api-access-rlwsz\") pod \"maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:11.782512 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.782446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4dd768c-253a-463f-bfb5-844bc5c5f243-tls-certs\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.782512 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.782482    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-dshm\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.782702 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.782527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmfcc\" (UniqueName: \"kubernetes.io/projected/d4dd768c-253a-463f-bfb5-844bc5c5f243-kube-api-access-mmfcc\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.782702 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.782545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-model-cache\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.782702 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.782589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-home\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.782887 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.782726    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-tmp-dir\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.783017 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.782995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-model-cache\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.783060 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.783000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-home\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.783123 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.783103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-tmp-dir\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.784886 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.784857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-dshm\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.785010 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.784994    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4dd768c-253a-463f-bfb5-844bc5c5f243-tls-certs\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.791534 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.791513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmfcc\" (UniqueName: \"kubernetes.io/projected/d4dd768c-253a-463f-bfb5-844bc5c5f243-kube-api-access-mmfcc\") pod \"test-model-0777c1-kserve-6f556d9847-n8l8k\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") " pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:11.804501 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.804473    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cbc1301-3425-4ef9-952e-9347fbfae04d" path="/var/lib/kubelet/pods/7cbc1301-3425-4ef9-952e-9347fbfae04d/volumes"
Sep 14 11:07:11.888251 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:11.888181    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:12.177102 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:12.177072    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"]
Sep 14 11:07:12.178462 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:07:12.178438    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4dd768c_253a_463f_bfb5_844bc5c5f243.slice/crio-6d8c8c56cbd3e45205e17b66483b7120cae60d639fba9855c6fe55bd47f89654 WatchSource:0}: Error finding container 6d8c8c56cbd3e45205e17b66483b7120cae60d639fba9855c6fe55bd47f89654: Status 404 returned error can't find the container with id 6d8c8c56cbd3e45205e17b66483b7120cae60d639fba9855c6fe55bd47f89654
Sep 14 11:07:12.186489 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:12.186465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-maas-api-tls\") pod \"maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:12.188963 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:12.188933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-maas-api-tls\") pod \"maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:12.374770 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:12.374735    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:12.661538 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:12.661515    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"]
Sep 14 11:07:12.662724 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:07:12.662688    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc02c3e9_15eb_4b37_af4c_ad2730e44e0e.slice/crio-a05ec0dbaecbc122c266af15f1d466a9b5ec2062ddc840076f723d7f324d74ac WatchSource:0}: Error finding container a05ec0dbaecbc122c266af15f1d466a9b5ec2062ddc840076f723d7f324d74ac: Status 404 returned error can't find the container with id a05ec0dbaecbc122c266af15f1d466a9b5ec2062ddc840076f723d7f324d74ac
Sep 14 11:07:12.931148 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:12.931107    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k" event={"ID":"d4dd768c-253a-463f-bfb5-844bc5c5f243","Type":"ContainerStarted","Data":"9eda1dc7b9e5132cd58b4a06526f14a250fbdcefeb6e07b51efd28ccbf1687e3"}
Sep 14 11:07:12.931346 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:12.931159    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k" event={"ID":"d4dd768c-253a-463f-bfb5-844bc5c5f243","Type":"ContainerStarted","Data":"6d8c8c56cbd3e45205e17b66483b7120cae60d639fba9855c6fe55bd47f89654"}
Sep 14 11:07:12.932461 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:12.932436    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9" event={"ID":"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e","Type":"ContainerStarted","Data":"a05ec0dbaecbc122c266af15f1d466a9b5ec2062ddc840076f723d7f324d74ac"}
Sep 14 11:07:12.955762 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:12.955691    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k" podStartSLOduration=1.9556706940000002 podStartE2EDuration="1.955670694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:07:12.952086715 +0000 UTC m=+1137.636771107" watchObservedRunningTime="2026-09-14 11:07:12.955670694 +0000 UTC m=+1137.640355085"
Sep 14 11:07:13.940559 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:13.940517    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9" event={"ID":"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e","Type":"ContainerStarted","Data":"f45146de92ba41242471440fe910e8512b4703b92c6fff9c761144b7d5de1862"}
Sep 14 11:07:13.940998 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:13.940840    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:13.963293 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:13.963243    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9" podStartSLOduration=2.338552205 podStartE2EDuration="2.963227501s" totalImagesPullingTime="624.675296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:07:13.95996187 +0000 UTC m=+1138.644646283" watchObservedRunningTime="2026-09-14 11:07:13.963227501 +0000 UTC m=+1138.647911936"
Sep 14 11:07:20.959935 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:20.959905    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:07:21.889194 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:21.889159    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:21.889377 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:21.889235    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:21.901735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:21.901691    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:22.000810 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:22.000781    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:07:31.340651 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:31.340613    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:07:48.739672 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:07:48.739591    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:08:08.322890 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:08.322854    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:08:12.748561 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.748524    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"]
Sep 14 11:08:12.754725 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.754692    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.758233 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.758191    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-cc14fe\"/\"kube-root-ca.crt\""
Sep 14 11:08:12.759402 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.759382    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-cc14fe\"/\"openshift-service-ca.crt\""
Sep 14 11:08:12.759521 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.759384    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-cc14fe\"/\"explicit-model-cc14fe-kserve-self-signed-certs\""
Sep 14 11:08:12.760614 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.760588    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-cc14fe\"/\"default-dockercfg-4zvwh\""
Sep 14 11:08:12.767956 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.767934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-dshm\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.768074 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.767986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gkjq\" (UniqueName: \"kubernetes.io/projected/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-kube-api-access-9gkjq\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.768141 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.768095    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-tls-certs\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.768197 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.768155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-model-cache\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.768257 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.768205    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-home\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.768257 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.768249    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-tmp-dir\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.771284 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.771259    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"]
Sep 14 11:08:12.869124 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.869094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-tmp-dir\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.869317 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.869148    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-dshm\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.869389 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.869331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gkjq\" (UniqueName: \"kubernetes.io/projected/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-kube-api-access-9gkjq\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.869453 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.869429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-tls-certs\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.869511 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.869501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-model-cache\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.869628 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.869609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-home\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.869964 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.869932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-tmp-dir\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.870089 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.869938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-model-cache\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.870089 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.870016    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-home\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.871517 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.871497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-dshm\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.871956 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.871940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-tls-certs\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:12.881536 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:12.881515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gkjq\" (UniqueName: \"kubernetes.io/projected/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-kube-api-access-9gkjq\") pod \"explicit-model-cc14fe-kserve-686b4bf66f-kfqdr\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") " pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:13.065044 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:13.064946    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:13.352132 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:13.352066    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"]
Sep 14 11:08:13.354491 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:08:13.354463    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd66114f4_7f00_4c3f_b430_25a03b2d8dcb.slice/crio-e57def0ed5e49b62a6d448e86a99d760699eb60a2b5afcb004dc2feaea86c91d WatchSource:0}: Error finding container e57def0ed5e49b62a6d448e86a99d760699eb60a2b5afcb004dc2feaea86c91d: Status 404 returned error can't find the container with id e57def0ed5e49b62a6d448e86a99d760699eb60a2b5afcb004dc2feaea86c91d
Sep 14 11:08:14.322095 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:14.322055    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr" event={"ID":"d66114f4-7f00-4c3f-b430-25a03b2d8dcb","Type":"ContainerStarted","Data":"357c870011edd16b3e44b3a65c01562436910c525b1b072e954caa774b937ea8"}
Sep 14 11:08:14.322472 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:14.322106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr" event={"ID":"d66114f4-7f00-4c3f-b430-25a03b2d8dcb","Type":"ContainerStarted","Data":"e57def0ed5e49b62a6d448e86a99d760699eb60a2b5afcb004dc2feaea86c91d"}
Sep 14 11:08:14.352706 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:14.352653    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr" podStartSLOduration=2.352633358 podStartE2EDuration="2.352633358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:08:14.350477821 +0000 UTC m=+1199.035162209" watchObservedRunningTime="2026-09-14 11:08:14.352633358 +0000 UTC m=+1199.037317771"
Sep 14 11:08:15.881803 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:15.881770    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 11:08:15.882264 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:15.881770    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 11:08:23.065434 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:23.065399    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:23.065891 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:23.065462    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:23.078218 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:23.078193    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:23.389059 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:23.389029    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:33.575090 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:33.575054    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"]
Sep 14 11:08:38.620019 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:38.619978    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:08:45.382731 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.382678    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"]
Sep 14 11:08:45.389584 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.389563    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.392474 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.392447    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"e2e-conflict-b-3c60bd48-kserve-self-signed-certs\""
Sep 14 11:08:45.405227 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.405203    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"]
Sep 14 11:08:45.487039 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.487009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-home\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.487172 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.487074    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-model-cache\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.487172 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.487098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-dshm\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.487172 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.487136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcv7q\" (UniqueName: \"kubernetes.io/projected/176b1345-f5c9-49b3-abf3-9d649fc92da7-kube-api-access-dcv7q\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.487172 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.487155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-tmp-dir\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.487310 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.487177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176b1345-f5c9-49b3-abf3-9d649fc92da7-tls-certs\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.587844 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.587814    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-home\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.588021 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.587893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-model-cache\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.588021 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.587927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-dshm\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.588117 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.588074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcv7q\" (UniqueName: \"kubernetes.io/projected/176b1345-f5c9-49b3-abf3-9d649fc92da7-kube-api-access-dcv7q\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.588172 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.588131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-tmp-dir\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.588232 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.588179    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176b1345-f5c9-49b3-abf3-9d649fc92da7-tls-certs\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.588282 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.588232    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-home\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.588282 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.588261    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-model-cache\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.588474 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.588452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-tmp-dir\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.590174 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.590155    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-dshm\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.590412 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.590397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176b1345-f5c9-49b3-abf3-9d649fc92da7-tls-certs\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.611255 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.611231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcv7q\" (UniqueName: \"kubernetes.io/projected/176b1345-f5c9-49b3-abf3-9d649fc92da7-kube-api-access-dcv7q\") pod \"e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.699317 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.699248    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:45.983572 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:45.983546    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"]
Sep 14 11:08:45.984681 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:08:45.984658    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod176b1345_f5c9_49b3_abf3_9d649fc92da7.slice/crio-1a090a68a9c400b9123e5fb12f1a2eca0343baa075fa8081272189bf0ce59465 WatchSource:0}: Error finding container 1a090a68a9c400b9123e5fb12f1a2eca0343baa075fa8081272189bf0ce59465: Status 404 returned error can't find the container with id 1a090a68a9c400b9123e5fb12f1a2eca0343baa075fa8081272189bf0ce59465
Sep 14 11:08:46.338445 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.338358    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"]
Sep 14 11:08:46.338631 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.338602    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9" podUID="bc02c3e9-15eb-4b37-af4c-ad2730e44e0e" containerName="maas-api" containerID="cri-o://f45146de92ba41242471440fe910e8512b4703b92c6fff9c761144b7d5de1862" gracePeriod=30
Sep 14 11:08:46.522469 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.522435    2561 generic.go:356] "Generic (PLEG): container finished" podID="bc02c3e9-15eb-4b37-af4c-ad2730e44e0e" containerID="f45146de92ba41242471440fe910e8512b4703b92c6fff9c761144b7d5de1862" exitCode=0
Sep 14 11:08:46.522852 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.522503    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9" event={"ID":"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e","Type":"ContainerDied","Data":"f45146de92ba41242471440fe910e8512b4703b92c6fff9c761144b7d5de1862"}
Sep 14 11:08:46.524287 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.524262    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w" event={"ID":"176b1345-f5c9-49b3-abf3-9d649fc92da7","Type":"ContainerStarted","Data":"317e38a4b8b6da1c4b5ea4b28311fd27c7f12eb09a903cfa3938fb5243d00709"}
Sep 14 11:08:46.524404 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.524303    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w" event={"ID":"176b1345-f5c9-49b3-abf3-9d649fc92da7","Type":"ContainerStarted","Data":"1a090a68a9c400b9123e5fb12f1a2eca0343baa075fa8081272189bf0ce59465"}
Sep 14 11:08:46.557353 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.557287    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w" podStartSLOduration=1.557266308 podStartE2EDuration="1.557266308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:08:46.553422797 +0000 UTC m=+1231.238107187" watchObservedRunningTime="2026-09-14 11:08:46.557266308 +0000 UTC m=+1231.241950715"
Sep 14 11:08:46.583782 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.583755    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:08:46.598508 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.598481    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-metrics-certs\") pod \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") "
Sep 14 11:08:46.598627 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.598614    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-maas-api-tls\") pod \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") "
Sep 14 11:08:46.598694 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.598652    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-openshift-service-ca\") pod \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") "
Sep 14 11:08:46.598775 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.598688    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlwsz\" (UniqueName: \"kubernetes.io/projected/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-kube-api-access-rlwsz\") pod \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\" (UID: \"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e\") "
Sep 14 11:08:46.599162 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.599102    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc02c3e9-15eb-4b37-af4c-ad2730e44e0e" (UID: "bc02c3e9-15eb-4b37-af4c-ad2730e44e0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:08:46.599842 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.599816    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:08:46.600975 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.600952    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bc02c3e9-15eb-4b37-af4c-ad2730e44e0e" (UID: "bc02c3e9-15eb-4b37-af4c-ad2730e44e0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:08:46.601135 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.601100    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc02c3e9-15eb-4b37-af4c-ad2730e44e0e" (UID: "bc02c3e9-15eb-4b37-af4c-ad2730e44e0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:08:46.601279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.601258    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-kube-api-access-rlwsz" (OuterVolumeSpecName: "kube-api-access-rlwsz") pod "bc02c3e9-15eb-4b37-af4c-ad2730e44e0e" (UID: "bc02c3e9-15eb-4b37-af4c-ad2730e44e0e"). InnerVolumeSpecName "kube-api-access-rlwsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:08:46.700822 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.700789    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:08:46.700822 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.700817    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:08:46.700822 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.700826    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlwsz\" (UniqueName: \"kubernetes.io/projected/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e-kube-api-access-rlwsz\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:08:46.747364 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.747327    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"]
Sep 14 11:08:46.748037 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.748007    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc02c3e9-15eb-4b37-af4c-ad2730e44e0e" containerName="maas-api"
Sep 14 11:08:46.748037 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.748029    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc02c3e9-15eb-4b37-af4c-ad2730e44e0e" containerName="maas-api"
Sep 14 11:08:46.748241 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.748150    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc02c3e9-15eb-4b37-af4c-ad2730e44e0e" containerName="maas-api"
Sep 14 11:08:46.751735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.751690    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:46.754689 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.754667    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e977dc37-serving-cert\""
Sep 14 11:08:46.794726 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.794686    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"]
Sep 14 11:08:46.801857 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.801831    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-metrics-certs\") pod \"maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:46.801969 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.801886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-maas-api-tls\") pod \"maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:46.801969 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.801955    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c222daf-9d8a-4bff-9266-9e849d283ac5-openshift-service-ca\") pod \"maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:46.802065 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.802000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp8td\" (UniqueName: \"kubernetes.io/projected/9c222daf-9d8a-4bff-9266-9e849d283ac5-kube-api-access-kp8td\") pod \"maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:46.904502 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.904246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-metrics-certs\") pod \"maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:46.904502 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.904323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-maas-api-tls\") pod \"maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:46.904502 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.904363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c222daf-9d8a-4bff-9266-9e849d283ac5-openshift-service-ca\") pod \"maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:46.904502 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.904421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp8td\" (UniqueName: \"kubernetes.io/projected/9c222daf-9d8a-4bff-9266-9e849d283ac5-kube-api-access-kp8td\") pod \"maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:46.904919 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:08:46.904777    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-serving-cert: secret "maas-api-e2e-mt-e977dc37-serving-cert" not found
Sep 14 11:08:46.904919 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:08:46.904852    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-maas-api-tls podName:9c222daf-9d8a-4bff-9266-9e849d283ac5 nodeName:}" failed. No retries permitted until 2026-09-14 11:08:47.404827758 +0000 UTC m=+1232.089512144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-maas-api-tls") pod "maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz" (UID: "9c222daf-9d8a-4bff-9266-9e849d283ac5") : secret "maas-api-e2e-mt-e977dc37-serving-cert" not found
Sep 14 11:08:46.905538 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.905480    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c222daf-9d8a-4bff-9266-9e849d283ac5-openshift-service-ca\") pod \"maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:46.907897 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.907873    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-metrics-certs\") pod \"maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:46.913806 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:46.913781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp8td\" (UniqueName: \"kubernetes.io/projected/9c222daf-9d8a-4bff-9266-9e849d283ac5-kube-api-access-kp8td\") pod \"maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:47.412246 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:47.412214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-maas-api-tls\") pod \"maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:47.414724 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:47.414690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-maas-api-tls\") pod \"maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:47.531660 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:47.531634    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"
Sep 14 11:08:47.531660 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:47.531648    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9" event={"ID":"bc02c3e9-15eb-4b37-af4c-ad2730e44e0e","Type":"ContainerDied","Data":"a05ec0dbaecbc122c266af15f1d466a9b5ec2062ddc840076f723d7f324d74ac"}
Sep 14 11:08:47.532134 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:47.531753    2561 scope.go:117] "RemoveContainer" containerID="f45146de92ba41242471440fe910e8512b4703b92c6fff9c761144b7d5de1862"
Sep 14 11:08:47.560911 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:47.560887    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"]
Sep 14 11:08:47.563625 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:47.563604    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cc14fe-5b6f94f769-rmzc9"]
Sep 14 11:08:47.666353 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:47.666277    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:47.803514 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:47.803481    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc02c3e9-15eb-4b37-af4c-ad2730e44e0e" path="/var/lib/kubelet/pods/bc02c3e9-15eb-4b37-af4c-ad2730e44e0e/volumes"
Sep 14 11:08:47.948256 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:47.948229    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"]
Sep 14 11:08:47.949072 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:08:47.949049    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c222daf_9d8a_4bff_9266_9e849d283ac5.slice/crio-803a5914909c61453cf51bbac5433481567f2272045f0daa118ac0ab48c86ec8 WatchSource:0}: Error finding container 803a5914909c61453cf51bbac5433481567f2272045f0daa118ac0ab48c86ec8: Status 404 returned error can't find the container with id 803a5914909c61453cf51bbac5433481567f2272045f0daa118ac0ab48c86ec8
Sep 14 11:08:47.950971 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:47.950954    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 11:08:48.539678 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.539638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz" event={"ID":"9c222daf-9d8a-4bff-9266-9e849d283ac5","Type":"ContainerStarted","Data":"803a5914909c61453cf51bbac5433481567f2272045f0daa118ac0ab48c86ec8"}
Sep 14 11:08:48.594140 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.594104    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr" podUID="d66114f4-7f00-4c3f-b430-25a03b2d8dcb" containerName="main" containerID="cri-o://357c870011edd16b3e44b3a65c01562436910c525b1b072e954caa774b937ea8" gracePeriod=45
Sep 14 11:08:48.855912 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.855883    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:48.928920 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.928884    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-tls-certs\") pod \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") "
Sep 14 11:08:48.929081 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.928944    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-home\") pod \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") "
Sep 14 11:08:48.929081 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.928973    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-dshm\") pod \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") "
Sep 14 11:08:48.929081 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.929040    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-model-cache\") pod \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") "
Sep 14 11:08:48.929253 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.929109    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-tmp-dir\") pod \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") "
Sep 14 11:08:48.929253 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.929153    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gkjq\" (UniqueName: \"kubernetes.io/projected/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-kube-api-access-9gkjq\") pod \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\" (UID: \"d66114f4-7f00-4c3f-b430-25a03b2d8dcb\") "
Sep 14 11:08:48.929371 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.929244    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-home" (OuterVolumeSpecName: "home") pod "d66114f4-7f00-4c3f-b430-25a03b2d8dcb" (UID: "d66114f4-7f00-4c3f-b430-25a03b2d8dcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:08:48.929371 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.929340    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d66114f4-7f00-4c3f-b430-25a03b2d8dcb" (UID: "d66114f4-7f00-4c3f-b430-25a03b2d8dcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:08:48.929371 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.929351    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "d66114f4-7f00-4c3f-b430-25a03b2d8dcb" (UID: "d66114f4-7f00-4c3f-b430-25a03b2d8dcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:08:48.929591 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.929566    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-tmp-dir\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:08:48.929674 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.929594    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-home\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:08:48.929674 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.929608    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-model-cache\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:08:48.931217 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.931185    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-dshm" (OuterVolumeSpecName: "dshm") pod "d66114f4-7f00-4c3f-b430-25a03b2d8dcb" (UID: "d66114f4-7f00-4c3f-b430-25a03b2d8dcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:08:48.931312 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.931225    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-kube-api-access-9gkjq" (OuterVolumeSpecName: "kube-api-access-9gkjq") pod "d66114f4-7f00-4c3f-b430-25a03b2d8dcb" (UID: "d66114f4-7f00-4c3f-b430-25a03b2d8dcb"). InnerVolumeSpecName "kube-api-access-9gkjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:08:48.931312 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:48.931277    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d66114f4-7f00-4c3f-b430-25a03b2d8dcb" (UID: "d66114f4-7f00-4c3f-b430-25a03b2d8dcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:08:49.030423 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.030395    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-tls-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:08:49.030423 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.030418    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-dshm\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:08:49.030423 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.030427    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gkjq\" (UniqueName: \"kubernetes.io/projected/d66114f4-7f00-4c3f-b430-25a03b2d8dcb-kube-api-access-9gkjq\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:08:49.547356 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.547311    2561 generic.go:356] "Generic (PLEG): container finished" podID="d66114f4-7f00-4c3f-b430-25a03b2d8dcb" containerID="357c870011edd16b3e44b3a65c01562436910c525b1b072e954caa774b937ea8" exitCode=0
Sep 14 11:08:49.547356 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.547350    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"
Sep 14 11:08:49.547883 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.547409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr" event={"ID":"d66114f4-7f00-4c3f-b430-25a03b2d8dcb","Type":"ContainerDied","Data":"357c870011edd16b3e44b3a65c01562436910c525b1b072e954caa774b937ea8"}
Sep 14 11:08:49.547883 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.547463    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr" event={"ID":"d66114f4-7f00-4c3f-b430-25a03b2d8dcb","Type":"ContainerDied","Data":"e57def0ed5e49b62a6d448e86a99d760699eb60a2b5afcb004dc2feaea86c91d"}
Sep 14 11:08:49.547883 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.547491    2561 scope.go:117] "RemoveContainer" containerID="357c870011edd16b3e44b3a65c01562436910c525b1b072e954caa774b937ea8"
Sep 14 11:08:49.549539 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.549509    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz" event={"ID":"9c222daf-9d8a-4bff-9266-9e849d283ac5","Type":"ContainerStarted","Data":"f855e1e5b21033e9b056aeba9d585b49d6006ebcec0ae14ed697dbd6f19f5923"}
Sep 14 11:08:49.549745 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.549601    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:49.558218 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.558202    2561 scope.go:117] "RemoveContainer" containerID="357c870011edd16b3e44b3a65c01562436910c525b1b072e954caa774b937ea8"
Sep 14 11:08:49.558477 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:08:49.558452    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357c870011edd16b3e44b3a65c01562436910c525b1b072e954caa774b937ea8\": container with ID starting with 357c870011edd16b3e44b3a65c01562436910c525b1b072e954caa774b937ea8 not found: ID does not exist" containerID="357c870011edd16b3e44b3a65c01562436910c525b1b072e954caa774b937ea8"
Sep 14 11:08:49.558551 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.558475    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357c870011edd16b3e44b3a65c01562436910c525b1b072e954caa774b937ea8"} err="failed to get container status \"357c870011edd16b3e44b3a65c01562436910c525b1b072e954caa774b937ea8\": rpc error: code = NotFound desc = could not find container \"357c870011edd16b3e44b3a65c01562436910c525b1b072e954caa774b937ea8\": container with ID starting with 357c870011edd16b3e44b3a65c01562436910c525b1b072e954caa774b937ea8 not found: ID does not exist"
Sep 14 11:08:49.577503 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.577458    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz" podStartSLOduration=2.145027335 podStartE2EDuration="3.577445347s" totalImagesPullingTime="1.432418012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:08:49.574200333 +0000 UTC m=+1234.258884722" watchObservedRunningTime="2026-09-14 11:08:49.577445347 +0000 UTC m=+1234.262129736"
Sep 14 11:08:49.591965 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.591941    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"]
Sep 14 11:08:49.597046 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.597020    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-cc14fe/explicit-model-cc14fe-kserve-686b4bf66f-kfqdr"]
Sep 14 11:08:49.804230 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:49.804148    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66114f4-7f00-4c3f-b430-25a03b2d8dcb" path="/var/lib/kubelet/pods/d66114f4-7f00-4c3f-b430-25a03b2d8dcb/volumes"
Sep 14 11:08:55.251817 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.251783    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"]
Sep 14 11:08:55.252477 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.252446    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66114f4-7f00-4c3f-b430-25a03b2d8dcb" containerName="main"
Sep 14 11:08:55.252477 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.252470    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66114f4-7f00-4c3f-b430-25a03b2d8dcb" containerName="main"
Sep 14 11:08:55.252632 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.252599    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66114f4-7f00-4c3f-b430-25a03b2d8dcb" containerName="main"
Sep 14 11:08:55.257340 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.257320    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.260087 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:08:55.260051    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="e77ecff7-edc4-466d-9fce-8e4ce49c494c" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.260230 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:08:55.260189    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-noten-d6e5fb-openshift-default-dockercfg-wpdl9\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-noten-d6e5fb-openshift-default-dockercfg-wpdl9\"" type="*v1.Secret"
Sep 14 11:08:55.275854 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.275831    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"]
Sep 14 11:08:55.396101 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.396062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istiod-ca-cert\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.396298 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.396133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-envoy\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.396298 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.396170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-workload-socket\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.396298 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.396200    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-credential-socket\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.396298 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.396269    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-data\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.396466 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.396321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-workload-certs\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.396466 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.396357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk9q8\" (UniqueName: \"kubernetes.io/projected/e77ecff7-edc4-466d-9fce-8e4ce49c494c-kube-api-access-pk9q8\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.396466 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.396390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-token\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.396466 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.396442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-podinfo\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.497015 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.496980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-workload-certs\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.497015 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.497017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk9q8\" (UniqueName: \"kubernetes.io/projected/e77ecff7-edc4-466d-9fce-8e4ce49c494c-kube-api-access-pk9q8\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.497270 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.497059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-token\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.497270 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.497099    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-podinfo\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.497270 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.497166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istiod-ca-cert\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.497270 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.497225    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-envoy\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.497270 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.497262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-workload-socket\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.497544 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.497297    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-credential-socket\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.497544 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.497362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-data\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.497544 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.497451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-workload-certs\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.497679 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.497579    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-workload-socket\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.497679 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.497648    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-data\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.497783 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.497758    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-credential-socket\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.498004 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.497984    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istiod-ca-cert\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.499608 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.499579    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-envoy\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.499831 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.499813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-podinfo\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.512495 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.512436    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk9q8\" (UniqueName: \"kubernetes.io/projected/e77ecff7-edc4-466d-9fce-8e4ce49c494c-kube-api-access-pk9q8\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.512594 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.512578    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-token\") pod \"e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") " pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:55.563173 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.563146    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:08:55.699451 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.699415    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:55.699647 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.699468    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:55.711726 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:55.711687    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:56.415805 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:56.415774    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-d6e5fb-openshift-default-dockercfg-wpdl9\""
Sep 14 11:08:56.419123 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:56.419104    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:56.613894 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:56.613864    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:08:56.755382 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:56.755295    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"]
Sep 14 11:08:56.778865 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:08:56.778837    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77ecff7_edc4_466d_9fce_8e4ce49c494c.slice/crio-949fb1bc24f687f9e0dc6fc6b3d05cebb901338d0ac9dd71d3e6255f95b1a410 WatchSource:0}: Error finding container 949fb1bc24f687f9e0dc6fc6b3d05cebb901338d0ac9dd71d3e6255f95b1a410: Status 404 returned error can't find the container with id 949fb1bc24f687f9e0dc6fc6b3d05cebb901338d0ac9dd71d3e6255f95b1a410
Sep 14 11:08:56.781215 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:56.781182    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:08:56.781327 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:56.781258    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:08:56.781327 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:56.781303    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:08:57.603607 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:57.603564    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t" event={"ID":"e77ecff7-edc4-466d-9fce-8e4ce49c494c","Type":"ContainerStarted","Data":"a74af7e6631e77820f3fdb4a5f53d73e39f954e6b2e192f0075b89e8fc5fb048"}
Sep 14 11:08:57.604087 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:57.603621    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t" event={"ID":"e77ecff7-edc4-466d-9fce-8e4ce49c494c","Type":"ContainerStarted","Data":"949fb1bc24f687f9e0dc6fc6b3d05cebb901338d0ac9dd71d3e6255f95b1a410"}
Sep 14 11:08:57.628845 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:57.628798    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t" podStartSLOduration=2.6287824410000002 podStartE2EDuration="2.628782441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:08:57.627369962 +0000 UTC m=+1242.312054375" watchObservedRunningTime="2026-09-14 11:08:57.628782441 +0000 UTC m=+1242.313466829"
Sep 14 11:08:58.419802 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:58.419763    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:58.425944 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:58.425914    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:58.614163 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:58.614129    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:08:58.615049 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:08:58.615027    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:09:14.673182 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:14.673102    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:09:25.269283 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:25.269245    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"]
Sep 14 11:09:30.833289 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:30.833252    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"]
Sep 14 11:09:30.835639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:30.833494    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz" podUID="9c222daf-9d8a-4bff-9266-9e849d283ac5" containerName="maas-api" containerID="cri-o://f855e1e5b21033e9b056aeba9d585b49d6006ebcec0ae14ed697dbd6f19f5923" gracePeriod=30
Sep 14 11:09:31.086336 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.086275    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:09:31.135226 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.135195    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-maas-api-tls\") pod \"9c222daf-9d8a-4bff-9266-9e849d283ac5\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") "
Sep 14 11:09:31.135368 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.135248    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp8td\" (UniqueName: \"kubernetes.io/projected/9c222daf-9d8a-4bff-9266-9e849d283ac5-kube-api-access-kp8td\") pod \"9c222daf-9d8a-4bff-9266-9e849d283ac5\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") "
Sep 14 11:09:31.135368 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.135318    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c222daf-9d8a-4bff-9266-9e849d283ac5-openshift-service-ca\") pod \"9c222daf-9d8a-4bff-9266-9e849d283ac5\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") "
Sep 14 11:09:31.135368 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.135354    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-metrics-certs\") pod \"9c222daf-9d8a-4bff-9266-9e849d283ac5\" (UID: \"9c222daf-9d8a-4bff-9266-9e849d283ac5\") "
Sep 14 11:09:31.135698 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.135666    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c222daf-9d8a-4bff-9266-9e849d283ac5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c222daf-9d8a-4bff-9266-9e849d283ac5" (UID: "9c222daf-9d8a-4bff-9266-9e849d283ac5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:09:31.137267 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.137238    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c222daf-9d8a-4bff-9266-9e849d283ac5" (UID: "9c222daf-9d8a-4bff-9266-9e849d283ac5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:09:31.137388 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.137370    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c222daf-9d8a-4bff-9266-9e849d283ac5-kube-api-access-kp8td" (OuterVolumeSpecName: "kube-api-access-kp8td") pod "9c222daf-9d8a-4bff-9266-9e849d283ac5" (UID: "9c222daf-9d8a-4bff-9266-9e849d283ac5"). InnerVolumeSpecName "kube-api-access-kp8td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:09:31.137433 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.137367    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9c222daf-9d8a-4bff-9266-9e849d283ac5" (UID: "9c222daf-9d8a-4bff-9266-9e849d283ac5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:09:31.237080 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.237045    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp8td\" (UniqueName: \"kubernetes.io/projected/9c222daf-9d8a-4bff-9266-9e849d283ac5-kube-api-access-kp8td\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:09:31.237080 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.237077    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c222daf-9d8a-4bff-9266-9e849d283ac5-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:09:31.237262 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.237090    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:09:31.237262 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.237102    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c222daf-9d8a-4bff-9266-9e849d283ac5-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:09:31.814443 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.814411    2561 generic.go:356] "Generic (PLEG): container finished" podID="9c222daf-9d8a-4bff-9266-9e849d283ac5" containerID="f855e1e5b21033e9b056aeba9d585b49d6006ebcec0ae14ed697dbd6f19f5923" exitCode=0
Sep 14 11:09:31.814576 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.814461    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"
Sep 14 11:09:31.814576 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.814488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz" event={"ID":"9c222daf-9d8a-4bff-9266-9e849d283ac5","Type":"ContainerDied","Data":"f855e1e5b21033e9b056aeba9d585b49d6006ebcec0ae14ed697dbd6f19f5923"}
Sep 14 11:09:31.814576 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.814533    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz" event={"ID":"9c222daf-9d8a-4bff-9266-9e849d283ac5","Type":"ContainerDied","Data":"803a5914909c61453cf51bbac5433481567f2272045f0daa118ac0ab48c86ec8"}
Sep 14 11:09:31.814576 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.814556    2561 scope.go:117] "RemoveContainer" containerID="f855e1e5b21033e9b056aeba9d585b49d6006ebcec0ae14ed697dbd6f19f5923"
Sep 14 11:09:31.823748 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.823707    2561 scope.go:117] "RemoveContainer" containerID="f855e1e5b21033e9b056aeba9d585b49d6006ebcec0ae14ed697dbd6f19f5923"
Sep 14 11:09:31.824040 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:09:31.824013    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f855e1e5b21033e9b056aeba9d585b49d6006ebcec0ae14ed697dbd6f19f5923\": container with ID starting with f855e1e5b21033e9b056aeba9d585b49d6006ebcec0ae14ed697dbd6f19f5923 not found: ID does not exist" containerID="f855e1e5b21033e9b056aeba9d585b49d6006ebcec0ae14ed697dbd6f19f5923"
Sep 14 11:09:31.824085 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.824050    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f855e1e5b21033e9b056aeba9d585b49d6006ebcec0ae14ed697dbd6f19f5923"} err="failed to get container status \"f855e1e5b21033e9b056aeba9d585b49d6006ebcec0ae14ed697dbd6f19f5923\": rpc error: code = NotFound desc = could not find container \"f855e1e5b21033e9b056aeba9d585b49d6006ebcec0ae14ed697dbd6f19f5923\": container with ID starting with f855e1e5b21033e9b056aeba9d585b49d6006ebcec0ae14ed697dbd6f19f5923 not found: ID does not exist"
Sep 14 11:09:31.840737 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.839664    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"]
Sep 14 11:09:31.848097 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:31.848069    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e977dc37-76dd549dc-pdrwz"]
Sep 14 11:09:33.803528 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:33.803496    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c222daf-9d8a-4bff-9266-9e849d283ac5" path="/var/lib/kubelet/pods/9c222daf-9d8a-4bff-9266-9e849d283ac5/volumes"
Sep 14 11:09:40.288445 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.288399    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w" podUID="176b1345-f5c9-49b3-abf3-9d649fc92da7" containerName="main" containerID="cri-o://317e38a4b8b6da1c4b5ea4b28311fd27c7f12eb09a903cfa3938fb5243d00709" gracePeriod=45
Sep 14 11:09:40.542432 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.542373    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:09:40.628672 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.628643    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-tmp-dir\") pod \"176b1345-f5c9-49b3-abf3-9d649fc92da7\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") "
Sep 14 11:09:40.628823 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.628706    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176b1345-f5c9-49b3-abf3-9d649fc92da7-tls-certs\") pod \"176b1345-f5c9-49b3-abf3-9d649fc92da7\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") "
Sep 14 11:09:40.628823 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.628763    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcv7q\" (UniqueName: \"kubernetes.io/projected/176b1345-f5c9-49b3-abf3-9d649fc92da7-kube-api-access-dcv7q\") pod \"176b1345-f5c9-49b3-abf3-9d649fc92da7\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") "
Sep 14 11:09:40.628899 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.628843    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-model-cache\") pod \"176b1345-f5c9-49b3-abf3-9d649fc92da7\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") "
Sep 14 11:09:40.628949 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.628900    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-dshm\") pod \"176b1345-f5c9-49b3-abf3-9d649fc92da7\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") "
Sep 14 11:09:40.628949 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.628897    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "176b1345-f5c9-49b3-abf3-9d649fc92da7" (UID: "176b1345-f5c9-49b3-abf3-9d649fc92da7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:09:40.629027 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.628972    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-home\") pod \"176b1345-f5c9-49b3-abf3-9d649fc92da7\" (UID: \"176b1345-f5c9-49b3-abf3-9d649fc92da7\") "
Sep 14 11:09:40.629113 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.629091    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-model-cache" (OuterVolumeSpecName: "model-cache") pod "176b1345-f5c9-49b3-abf3-9d649fc92da7" (UID: "176b1345-f5c9-49b3-abf3-9d649fc92da7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:09:40.629279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.629257    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-home" (OuterVolumeSpecName: "home") pod "176b1345-f5c9-49b3-abf3-9d649fc92da7" (UID: "176b1345-f5c9-49b3-abf3-9d649fc92da7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:09:40.629346 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.629304    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-model-cache\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:09:40.629346 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.629324    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-tmp-dir\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:09:40.630985 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.630965    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-dshm" (OuterVolumeSpecName: "dshm") pod "176b1345-f5c9-49b3-abf3-9d649fc92da7" (UID: "176b1345-f5c9-49b3-abf3-9d649fc92da7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:09:40.631277 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.631259    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176b1345-f5c9-49b3-abf3-9d649fc92da7-kube-api-access-dcv7q" (OuterVolumeSpecName: "kube-api-access-dcv7q") pod "176b1345-f5c9-49b3-abf3-9d649fc92da7" (UID: "176b1345-f5c9-49b3-abf3-9d649fc92da7"). InnerVolumeSpecName "kube-api-access-dcv7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:09:40.631331 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.631274    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176b1345-f5c9-49b3-abf3-9d649fc92da7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "176b1345-f5c9-49b3-abf3-9d649fc92da7" (UID: "176b1345-f5c9-49b3-abf3-9d649fc92da7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:09:40.730109 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.730086    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-dshm\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:09:40.730109 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.730107    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176b1345-f5c9-49b3-abf3-9d649fc92da7-home\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:09:40.730252 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.730115    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176b1345-f5c9-49b3-abf3-9d649fc92da7-tls-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:09:40.730252 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.730125    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcv7q\" (UniqueName: \"kubernetes.io/projected/176b1345-f5c9-49b3-abf3-9d649fc92da7-kube-api-access-dcv7q\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:09:40.870871 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.870838    2561 generic.go:356] "Generic (PLEG): container finished" podID="176b1345-f5c9-49b3-abf3-9d649fc92da7" containerID="317e38a4b8b6da1c4b5ea4b28311fd27c7f12eb09a903cfa3938fb5243d00709" exitCode=0
Sep 14 11:09:40.871047 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.870883    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"
Sep 14 11:09:40.871047 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.870910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w" event={"ID":"176b1345-f5c9-49b3-abf3-9d649fc92da7","Type":"ContainerDied","Data":"317e38a4b8b6da1c4b5ea4b28311fd27c7f12eb09a903cfa3938fb5243d00709"}
Sep 14 11:09:40.871047 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.870954    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w" event={"ID":"176b1345-f5c9-49b3-abf3-9d649fc92da7","Type":"ContainerDied","Data":"1a090a68a9c400b9123e5fb12f1a2eca0343baa075fa8081272189bf0ce59465"}
Sep 14 11:09:40.871047 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.870973    2561 scope.go:117] "RemoveContainer" containerID="317e38a4b8b6da1c4b5ea4b28311fd27c7f12eb09a903cfa3938fb5243d00709"
Sep 14 11:09:40.881010 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.880992    2561 scope.go:117] "RemoveContainer" containerID="317e38a4b8b6da1c4b5ea4b28311fd27c7f12eb09a903cfa3938fb5243d00709"
Sep 14 11:09:40.881294 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:09:40.881276    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317e38a4b8b6da1c4b5ea4b28311fd27c7f12eb09a903cfa3938fb5243d00709\": container with ID starting with 317e38a4b8b6da1c4b5ea4b28311fd27c7f12eb09a903cfa3938fb5243d00709 not found: ID does not exist" containerID="317e38a4b8b6da1c4b5ea4b28311fd27c7f12eb09a903cfa3938fb5243d00709"
Sep 14 11:09:40.881355 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.881302    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317e38a4b8b6da1c4b5ea4b28311fd27c7f12eb09a903cfa3938fb5243d00709"} err="failed to get container status \"317e38a4b8b6da1c4b5ea4b28311fd27c7f12eb09a903cfa3938fb5243d00709\": rpc error: code = NotFound desc = could not find container \"317e38a4b8b6da1c4b5ea4b28311fd27c7f12eb09a903cfa3938fb5243d00709\": container with ID starting with 317e38a4b8b6da1c4b5ea4b28311fd27c7f12eb09a903cfa3938fb5243d00709 not found: ID does not exist"
Sep 14 11:09:40.895163 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.895136    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"]
Sep 14 11:09:40.898181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:40.898160    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-b-3c60bd48-kserve-6dd88c87cc-jsd6w"]
Sep 14 11:09:41.803116 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:41.803088    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="176b1345-f5c9-49b3-abf3-9d649fc92da7" path="/var/lib/kubelet/pods/176b1345-f5c9-49b3-abf3-9d649fc92da7/volumes"
Sep 14 11:09:49.933595 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:49.933557    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:09:54.790582 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:54.790549    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"]
Sep 14 11:09:54.790966 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:54.790841    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t" podUID="e77ecff7-edc4-466d-9fce-8e4ce49c494c" containerName="istio-proxy" containerID="cri-o://a74af7e6631e77820f3fdb4a5f53d73e39f954e6b2e192f0075b89e8fc5fb048" gracePeriod=30
Sep 14 11:09:58.616530 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:58.616490    2561 patch_prober.go:28] interesting pod/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 11:09:58.616948 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:58.616552    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t" podUID="e77ecff7-edc4-466d-9fce-8e4ce49c494c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 11:09:59.994509 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:59.994464    2561 generic.go:356] "Generic (PLEG): container finished" podID="e77ecff7-edc4-466d-9fce-8e4ce49c494c" containerID="a74af7e6631e77820f3fdb4a5f53d73e39f954e6b2e192f0075b89e8fc5fb048" exitCode=0
Sep 14 11:09:59.994897 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:09:59.994610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t" event={"ID":"e77ecff7-edc4-466d-9fce-8e4ce49c494c","Type":"ContainerDied","Data":"a74af7e6631e77820f3fdb4a5f53d73e39f954e6b2e192f0075b89e8fc5fb048"}
Sep 14 11:10:00.049657 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.049586    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:10:00.215520 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.215489    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-workload-certs\") pod \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") "
Sep 14 11:10:00.215728 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.215533    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-data\") pod \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") "
Sep 14 11:10:00.215728 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.215552    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istiod-ca-cert\") pod \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") "
Sep 14 11:10:00.215728 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.215576    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk9q8\" (UniqueName: \"kubernetes.io/projected/e77ecff7-edc4-466d-9fce-8e4ce49c494c-kube-api-access-pk9q8\") pod \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") "
Sep 14 11:10:00.215728 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.215602    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-token\") pod \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") "
Sep 14 11:10:00.215728 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.215640    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-credential-socket\") pod \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") "
Sep 14 11:10:00.215728 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.215679    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-envoy\") pod \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") "
Sep 14 11:10:00.216086 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.215708    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-workload-socket\") pod \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") "
Sep 14 11:10:00.216086 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.215781    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-podinfo\") pod \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\" (UID: \"e77ecff7-edc4-466d-9fce-8e4ce49c494c\") "
Sep 14 11:10:00.216086 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.215852    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e77ecff7-edc4-466d-9fce-8e4ce49c494c" (UID: "e77ecff7-edc4-466d-9fce-8e4ce49c494c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:00.216086 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.215860    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-data" (OuterVolumeSpecName: "istio-data") pod "e77ecff7-edc4-466d-9fce-8e4ce49c494c" (UID: "e77ecff7-edc4-466d-9fce-8e4ce49c494c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:00.216288 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.216094    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e77ecff7-edc4-466d-9fce-8e4ce49c494c" (UID: "e77ecff7-edc4-466d-9fce-8e4ce49c494c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:10:00.216288 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.216106    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e77ecff7-edc4-466d-9fce-8e4ce49c494c" (UID: "e77ecff7-edc4-466d-9fce-8e4ce49c494c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:00.216288 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.216194    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e77ecff7-edc4-466d-9fce-8e4ce49c494c" (UID: "e77ecff7-edc4-466d-9fce-8e4ce49c494c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:00.216288 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.216213    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-workload-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:00.216288 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.216232    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-data\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:00.216288 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.216246    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istiod-ca-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:00.216288 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.216259    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-workload-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:00.217812 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.217788    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77ecff7-edc4-466d-9fce-8e4ce49c494c-kube-api-access-pk9q8" (OuterVolumeSpecName: "kube-api-access-pk9q8") pod "e77ecff7-edc4-466d-9fce-8e4ce49c494c" (UID: "e77ecff7-edc4-466d-9fce-8e4ce49c494c"). InnerVolumeSpecName "kube-api-access-pk9q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:10:00.218029 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.218013    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e77ecff7-edc4-466d-9fce-8e4ce49c494c" (UID: "e77ecff7-edc4-466d-9fce-8e4ce49c494c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 11:10:00.218258 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.218234    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-token" (OuterVolumeSpecName: "istio-token") pod "e77ecff7-edc4-466d-9fce-8e4ce49c494c" (UID: "e77ecff7-edc4-466d-9fce-8e4ce49c494c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:10:00.218320 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.218236    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e77ecff7-edc4-466d-9fce-8e4ce49c494c" (UID: "e77ecff7-edc4-466d-9fce-8e4ce49c494c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:00.317072 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.317002    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-podinfo\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:00.317072 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.317027    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk9q8\" (UniqueName: \"kubernetes.io/projected/e77ecff7-edc4-466d-9fce-8e4ce49c494c-kube-api-access-pk9q8\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:00.317072 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.317038    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:00.317072 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.317047    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-credential-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:00.317072 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:00.317055    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77ecff7-edc4-466d-9fce-8e4ce49c494c-istio-envoy\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:01.002640 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:01.002606    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"
Sep 14 11:10:01.003109 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:01.002638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t" event={"ID":"e77ecff7-edc4-466d-9fce-8e4ce49c494c","Type":"ContainerDied","Data":"949fb1bc24f687f9e0dc6fc6b3d05cebb901338d0ac9dd71d3e6255f95b1a410"}
Sep 14 11:10:01.003109 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:01.002701    2561 scope.go:117] "RemoveContainer" containerID="a74af7e6631e77820f3fdb4a5f53d73e39f954e6b2e192f0075b89e8fc5fb048"
Sep 14 11:10:01.027331 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:01.027304    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"]
Sep 14 11:10:01.034728 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:01.033128    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-d6e5fb-openshift-default-9d978b84d-5kk9t"]
Sep 14 11:10:01.803826 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:01.803794    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e77ecff7-edc4-466d-9fce-8e4ce49c494c" path="/var/lib/kubelet/pods/e77ecff7-edc4-466d-9fce-8e4ce49c494c/volumes"
Sep 14 11:10:02.362420 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.362385    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"]
Sep 14 11:10:02.362969 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.362951    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176b1345-f5c9-49b3-abf3-9d649fc92da7" containerName="main"
Sep 14 11:10:02.363038 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.362972    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="176b1345-f5c9-49b3-abf3-9d649fc92da7" containerName="main"
Sep 14 11:10:02.363038 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.362991    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c222daf-9d8a-4bff-9266-9e849d283ac5" containerName="maas-api"
Sep 14 11:10:02.363038 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.362998    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c222daf-9d8a-4bff-9266-9e849d283ac5" containerName="maas-api"
Sep 14 11:10:02.363038 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.363008    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e77ecff7-edc4-466d-9fce-8e4ce49c494c" containerName="istio-proxy"
Sep 14 11:10:02.363038 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.363014    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e77ecff7-edc4-466d-9fce-8e4ce49c494c" containerName="istio-proxy"
Sep 14 11:10:02.363193 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.363083    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e77ecff7-edc4-466d-9fce-8e4ce49c494c" containerName="istio-proxy"
Sep 14 11:10:02.363193 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.363091    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c222daf-9d8a-4bff-9266-9e849d283ac5" containerName="maas-api"
Sep 14 11:10:02.363193 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.363100    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="176b1345-f5c9-49b3-abf3-9d649fc92da7" containerName="main"
Sep 14 11:10:02.367630 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.367611    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:02.371317 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.371293    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-ae9d47-serving-cert\""
Sep 14 11:10:02.384152 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.384128    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"]
Sep 14 11:10:02.536124 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.536089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hmlg\" (UniqueName: \"kubernetes.io/projected/67039613-681d-44e6-b719-ba5086dd7823-kube-api-access-7hmlg\") pod \"maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:02.536315 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.536158    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:02.536315 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.536220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:02.536315 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.536244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67039613-681d-44e6-b719-ba5086dd7823-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:02.631636 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.631563    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"]
Sep 14 11:10:02.637583 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.637553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:02.637744 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.637594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67039613-681d-44e6-b719-ba5086dd7823-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:02.637744 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.637661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hmlg\" (UniqueName: \"kubernetes.io/projected/67039613-681d-44e6-b719-ba5086dd7823-kube-api-access-7hmlg\") pod \"maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:02.637744 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:02.637729    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-serving-cert: secret "maas-api-e2e-shared-a-w2-ae9d47-serving-cert" not found
Sep 14 11:10:02.637970 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.637774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:02.637970 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:02.637810    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-maas-api-tls podName:67039613-681d-44e6-b719-ba5086dd7823 nodeName:}" failed. No retries permitted until 2026-09-14 11:10:03.13778869 +0000 UTC m=+1307.822473061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-maas-api-tls") pod "maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj" (UID: "67039613-681d-44e6-b719-ba5086dd7823") : secret "maas-api-e2e-shared-a-w2-ae9d47-serving-cert" not found
Sep 14 11:10:02.638337 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.638315    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67039613-681d-44e6-b719-ba5086dd7823-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:02.639978 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.639953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:02.649376 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:02.649352    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hmlg\" (UniqueName: \"kubernetes.io/projected/67039613-681d-44e6-b719-ba5086dd7823-kube-api-access-7hmlg\") pod \"maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:03.143570 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:03.143539    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:03.146016 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:03.145983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:03.278669 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:03.278634    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:03.567797 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:03.567769    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"]
Sep 14 11:10:03.568667 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:10:03.568642    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67039613_681d_44e6_b719_ba5086dd7823.slice/crio-b6ece2cbb2b7efb10c36fc610e485fd53d43d0d5a330944c201064c792ae4ac1 WatchSource:0}: Error finding container b6ece2cbb2b7efb10c36fc610e485fd53d43d0d5a330944c201064c792ae4ac1: Status 404 returned error can't find the container with id b6ece2cbb2b7efb10c36fc610e485fd53d43d0d5a330944c201064c792ae4ac1
Sep 14 11:10:04.023507 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:04.023473    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj" event={"ID":"67039613-681d-44e6-b719-ba5086dd7823","Type":"ContainerStarted","Data":"b6ece2cbb2b7efb10c36fc610e485fd53d43d0d5a330944c201064c792ae4ac1"}
Sep 14 11:10:05.030809 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:05.030685    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj" event={"ID":"67039613-681d-44e6-b719-ba5086dd7823","Type":"ContainerStarted","Data":"bb27bb8554d6c91d0fd03c5dab0c2e222371c06e9517bd9a8bb984bc69a160ab"}
Sep 14 11:10:05.030809 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:05.030777    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:05.057789 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:05.057738    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj" podStartSLOduration=1.9266318550000001 podStartE2EDuration="3.057697226s" totalImagesPullingTime="1.131065371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:05.054229889 +0000 UTC m=+1309.738914278" watchObservedRunningTime="2026-09-14 11:10:05.057697226 +0000 UTC m=+1309.742381616"
Sep 14 11:10:09.223060 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.223020    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"]
Sep 14 11:10:09.223478 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.223280    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb" podUID="bf7280d4-31a2-4608-89e5-ab9f979c1b0c" containerName="maas-api" containerID="cri-o://8de616025b932726d94564263c3501acc4f0c36beb6893b054500482160467c9" gracePeriod=30
Sep 14 11:10:09.469179 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.469156    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:10:09.507385 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.507302    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-metrics-certs\") pod \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") "
Sep 14 11:10:09.508026 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.507985    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-openshift-service-ca\") pod \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") "
Sep 14 11:10:09.508921 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.508801    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf7280d4-31a2-4608-89e5-ab9f979c1b0c" (UID: "bf7280d4-31a2-4608-89e5-ab9f979c1b0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:10:09.509426 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.509396    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-maas-api-tls\") pod \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") "
Sep 14 11:10:09.509626 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.509602    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hdmk\" (UniqueName: \"kubernetes.io/projected/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-kube-api-access-5hdmk\") pod \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\" (UID: \"bf7280d4-31a2-4608-89e5-ab9f979c1b0c\") "
Sep 14 11:10:09.510433 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.510378    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:09.513797 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.513770    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-kube-api-access-5hdmk" (OuterVolumeSpecName: "kube-api-access-5hdmk") pod "bf7280d4-31a2-4608-89e5-ab9f979c1b0c" (UID: "bf7280d4-31a2-4608-89e5-ab9f979c1b0c"). InnerVolumeSpecName "kube-api-access-5hdmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:10:09.514156 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.514125    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bf7280d4-31a2-4608-89e5-ab9f979c1b0c" (UID: "bf7280d4-31a2-4608-89e5-ab9f979c1b0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:10:09.514261 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.514223    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf7280d4-31a2-4608-89e5-ab9f979c1b0c" (UID: "bf7280d4-31a2-4608-89e5-ab9f979c1b0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:10:09.611406 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.611370    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hdmk\" (UniqueName: \"kubernetes.io/projected/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-kube-api-access-5hdmk\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:09.611406 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.611401    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:09.611406 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:09.611410    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7280d4-31a2-4608-89e5-ab9f979c1b0c-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:10.063353 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:10.063260    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf7280d4-31a2-4608-89e5-ab9f979c1b0c" containerID="8de616025b932726d94564263c3501acc4f0c36beb6893b054500482160467c9" exitCode=0
Sep 14 11:10:10.063353 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:10.063303    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"
Sep 14 11:10:10.063533 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:10.063347    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb" event={"ID":"bf7280d4-31a2-4608-89e5-ab9f979c1b0c","Type":"ContainerDied","Data":"8de616025b932726d94564263c3501acc4f0c36beb6893b054500482160467c9"}
Sep 14 11:10:10.063533 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:10.063401    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb" event={"ID":"bf7280d4-31a2-4608-89e5-ab9f979c1b0c","Type":"ContainerDied","Data":"56dcfeccfc68ce81fb4351e85913ab98e795698934813ee8eafd0aae3a16668e"}
Sep 14 11:10:10.063533 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:10.063426    2561 scope.go:117] "RemoveContainer" containerID="8de616025b932726d94564263c3501acc4f0c36beb6893b054500482160467c9"
Sep 14 11:10:10.072774 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:10.072752    2561 scope.go:117] "RemoveContainer" containerID="8de616025b932726d94564263c3501acc4f0c36beb6893b054500482160467c9"
Sep 14 11:10:10.073053 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:10.073032    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de616025b932726d94564263c3501acc4f0c36beb6893b054500482160467c9\": container with ID starting with 8de616025b932726d94564263c3501acc4f0c36beb6893b054500482160467c9 not found: ID does not exist" containerID="8de616025b932726d94564263c3501acc4f0c36beb6893b054500482160467c9"
Sep 14 11:10:10.073119 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:10.073061    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de616025b932726d94564263c3501acc4f0c36beb6893b054500482160467c9"} err="failed to get container status \"8de616025b932726d94564263c3501acc4f0c36beb6893b054500482160467c9\": rpc error: code = NotFound desc = could not find container \"8de616025b932726d94564263c3501acc4f0c36beb6893b054500482160467c9\": container with ID starting with 8de616025b932726d94564263c3501acc4f0c36beb6893b054500482160467c9 not found: ID does not exist"
Sep 14 11:10:10.119756 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:10.119726    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"]
Sep 14 11:10:10.124409 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:10.124387    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04195c-6544d95856-rlhgb"]
Sep 14 11:10:11.042796 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:11.042765    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:10:11.803872 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:11.803836    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf7280d4-31a2-4608-89e5-ab9f979c1b0c" path="/var/lib/kubelet/pods/bf7280d4-31a2-4608-89e5-ab9f979c1b0c/volumes"
Sep 14 11:10:15.337202 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:15.337165    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:10:16.655243 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.655206    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"]
Sep 14 11:10:16.655934 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.655913    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf7280d4-31a2-4608-89e5-ab9f979c1b0c" containerName="maas-api"
Sep 14 11:10:16.656012 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.655938    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf7280d4-31a2-4608-89e5-ab9f979c1b0c" containerName="maas-api"
Sep 14 11:10:16.656108 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.656088    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf7280d4-31a2-4608-89e5-ab9f979c1b0c" containerName="maas-api"
Sep 14 11:10:16.660959 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.660936    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.664277 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.664258    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-22b014-openshift-default-dockercfg-wph4t\""
Sep 14 11:10:16.674193 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.674169    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"]
Sep 14 11:10:16.786143 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.786098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-credential-socket\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.786333 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.786157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-istio-data\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.786333 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.786202    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-istio-envoy\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.786333 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.786229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e419258-d263-4df1-aaf0-422eace67cc9-istiod-ca-cert\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.786333 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.786266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-workload-certs\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.786333 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.786322    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-workload-socket\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.786704 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.786355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e419258-d263-4df1-aaf0-422eace67cc9-istio-podinfo\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.786704 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.786392    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvwsm\" (UniqueName: \"kubernetes.io/projected/9e419258-d263-4df1-aaf0-422eace67cc9-kube-api-access-cvwsm\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.786704 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.786447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e419258-d263-4df1-aaf0-422eace67cc9-istio-token\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.887540 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.887500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-credential-socket\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.887735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.887557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-istio-data\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.887735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.887636    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-istio-envoy\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.887735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.887667    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e419258-d263-4df1-aaf0-422eace67cc9-istiod-ca-cert\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.887735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.887706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-workload-certs\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.887990 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.887784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-workload-socket\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.887990 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.887822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e419258-d263-4df1-aaf0-422eace67cc9-istio-podinfo\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.887990 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.887887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvwsm\" (UniqueName: \"kubernetes.io/projected/9e419258-d263-4df1-aaf0-422eace67cc9-kube-api-access-cvwsm\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.887990 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.887975    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-credential-socket\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.888206 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.887979    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e419258-d263-4df1-aaf0-422eace67cc9-istio-token\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.888265 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.888209    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-istio-data\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.888640 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.888611    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-workload-socket\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.888967 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.888901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-workload-certs\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.889086 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.889016    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e419258-d263-4df1-aaf0-422eace67cc9-istiod-ca-cert\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.891120 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.891014    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-istio-envoy\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.891226 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.891196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e419258-d263-4df1-aaf0-422eace67cc9-istio-podinfo\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.896895 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.896868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e419258-d263-4df1-aaf0-422eace67cc9-istio-token\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.897001 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.896889    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvwsm\" (UniqueName: \"kubernetes.io/projected/9e419258-d263-4df1-aaf0-422eace67cc9-kube-api-access-cvwsm\") pod \"e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") " pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:16.972926 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:16.972839    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:17.322153 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:10:17.322117    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e419258_d263_4df1_aaf0_422eace67cc9.slice/crio-7c44e2c4cf7408296d8f7d6104a54617fe28b2a991488d1365a02bfeba903c20 WatchSource:0}: Error finding container 7c44e2c4cf7408296d8f7d6104a54617fe28b2a991488d1365a02bfeba903c20: Status 404 returned error can't find the container with id 7c44e2c4cf7408296d8f7d6104a54617fe28b2a991488d1365a02bfeba903c20
Sep 14 11:10:17.322380 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.322361    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"]
Sep 14 11:10:17.324556 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.324523    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:10:17.324662 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.324586    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:10:17.324662 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.324617    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:10:17.652195 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.652152    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k" podUID="d4dd768c-253a-463f-bfb5-844bc5c5f243" containerName="main" containerID="cri-o://9eda1dc7b9e5132cd58b4a06526f14a250fbdcefeb6e07b51efd28ccbf1687e3" gracePeriod=45
Sep 14 11:10:17.910149 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.910090    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:10:17.999168 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.999136    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-dshm\") pod \"d4dd768c-253a-463f-bfb5-844bc5c5f243\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") "
Sep 14 11:10:17.999287 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.999220    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-model-cache\") pod \"d4dd768c-253a-463f-bfb5-844bc5c5f243\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") "
Sep 14 11:10:17.999287 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.999273    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-home\") pod \"d4dd768c-253a-463f-bfb5-844bc5c5f243\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") "
Sep 14 11:10:17.999392 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.999319    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-tmp-dir\") pod \"d4dd768c-253a-463f-bfb5-844bc5c5f243\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") "
Sep 14 11:10:17.999392 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.999362    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmfcc\" (UniqueName: \"kubernetes.io/projected/d4dd768c-253a-463f-bfb5-844bc5c5f243-kube-api-access-mmfcc\") pod \"d4dd768c-253a-463f-bfb5-844bc5c5f243\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") "
Sep 14 11:10:17.999502 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.999392    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4dd768c-253a-463f-bfb5-844bc5c5f243-tls-certs\") pod \"d4dd768c-253a-463f-bfb5-844bc5c5f243\" (UID: \"d4dd768c-253a-463f-bfb5-844bc5c5f243\") "
Sep 14 11:10:17.999502 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.999482    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4dd768c-253a-463f-bfb5-844bc5c5f243" (UID: "d4dd768c-253a-463f-bfb5-844bc5c5f243"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:17.999600 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.999552    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4dd768c-253a-463f-bfb5-844bc5c5f243" (UID: "d4dd768c-253a-463f-bfb5-844bc5c5f243"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:17.999600 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.999570    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-home" (OuterVolumeSpecName: "home") pod "d4dd768c-253a-463f-bfb5-844bc5c5f243" (UID: "d4dd768c-253a-463f-bfb5-844bc5c5f243"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:17.999834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.999813    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-model-cache\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:17.999928 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.999838    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-home\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:17.999928 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:17.999851    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-tmp-dir\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:18.001405 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.001381    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-dshm" (OuterVolumeSpecName: "dshm") pod "d4dd768c-253a-463f-bfb5-844bc5c5f243" (UID: "d4dd768c-253a-463f-bfb5-844bc5c5f243"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:18.001506 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.001478    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4dd768c-253a-463f-bfb5-844bc5c5f243-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4dd768c-253a-463f-bfb5-844bc5c5f243" (UID: "d4dd768c-253a-463f-bfb5-844bc5c5f243"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:10:18.001549 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.001534    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4dd768c-253a-463f-bfb5-844bc5c5f243-kube-api-access-mmfcc" (OuterVolumeSpecName: "kube-api-access-mmfcc") pod "d4dd768c-253a-463f-bfb5-844bc5c5f243" (UID: "d4dd768c-253a-463f-bfb5-844bc5c5f243"). InnerVolumeSpecName "kube-api-access-mmfcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:10:18.100415 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.100372    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4dd768c-253a-463f-bfb5-844bc5c5f243-dshm\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:18.100415 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.100409    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmfcc\" (UniqueName: \"kubernetes.io/projected/d4dd768c-253a-463f-bfb5-844bc5c5f243-kube-api-access-mmfcc\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:18.100619 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.100425    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4dd768c-253a-463f-bfb5-844bc5c5f243-tls-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:18.116792 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.116758    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4dd768c-253a-463f-bfb5-844bc5c5f243" containerID="9eda1dc7b9e5132cd58b4a06526f14a250fbdcefeb6e07b51efd28ccbf1687e3" exitCode=0
Sep 14 11:10:18.116946 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.116794    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k" event={"ID":"d4dd768c-253a-463f-bfb5-844bc5c5f243","Type":"ContainerDied","Data":"9eda1dc7b9e5132cd58b4a06526f14a250fbdcefeb6e07b51efd28ccbf1687e3"}
Sep 14 11:10:18.116946 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.116812    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"
Sep 14 11:10:18.116946 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.116868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k" event={"ID":"d4dd768c-253a-463f-bfb5-844bc5c5f243","Type":"ContainerDied","Data":"6d8c8c56cbd3e45205e17b66483b7120cae60d639fba9855c6fe55bd47f89654"}
Sep 14 11:10:18.116946 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.116894    2561 scope.go:117] "RemoveContainer" containerID="9eda1dc7b9e5132cd58b4a06526f14a250fbdcefeb6e07b51efd28ccbf1687e3"
Sep 14 11:10:18.118995 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.118969    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv" event={"ID":"9e419258-d263-4df1-aaf0-422eace67cc9","Type":"ContainerStarted","Data":"71db1e8a767f07cfb39c07d2dd500d7a61a3b6deb07b70eddec204826147ad90"}
Sep 14 11:10:18.119126 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.119024    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv" event={"ID":"9e419258-d263-4df1-aaf0-422eace67cc9","Type":"ContainerStarted","Data":"7c44e2c4cf7408296d8f7d6104a54617fe28b2a991488d1365a02bfeba903c20"}
Sep 14 11:10:18.127642 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.127621    2561 scope.go:117] "RemoveContainer" containerID="9eda1dc7b9e5132cd58b4a06526f14a250fbdcefeb6e07b51efd28ccbf1687e3"
Sep 14 11:10:18.127953 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:18.127934    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eda1dc7b9e5132cd58b4a06526f14a250fbdcefeb6e07b51efd28ccbf1687e3\": container with ID starting with 9eda1dc7b9e5132cd58b4a06526f14a250fbdcefeb6e07b51efd28ccbf1687e3 not found: ID does not exist" containerID="9eda1dc7b9e5132cd58b4a06526f14a250fbdcefeb6e07b51efd28ccbf1687e3"
Sep 14 11:10:18.128036 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.127963    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eda1dc7b9e5132cd58b4a06526f14a250fbdcefeb6e07b51efd28ccbf1687e3"} err="failed to get container status \"9eda1dc7b9e5132cd58b4a06526f14a250fbdcefeb6e07b51efd28ccbf1687e3\": rpc error: code = NotFound desc = could not find container \"9eda1dc7b9e5132cd58b4a06526f14a250fbdcefeb6e07b51efd28ccbf1687e3\": container with ID starting with 9eda1dc7b9e5132cd58b4a06526f14a250fbdcefeb6e07b51efd28ccbf1687e3 not found: ID does not exist"
Sep 14 11:10:18.149566 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.149507    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv" podStartSLOduration=2.149491452 podStartE2EDuration="2.149491452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:18.145669704 +0000 UTC m=+1322.830354087" watchObservedRunningTime="2026-09-14 11:10:18.149491452 +0000 UTC m=+1322.834175857"
Sep 14 11:10:18.165221 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.165166    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"]
Sep 14 11:10:18.167652 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.167631    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0777c1/test-model-0777c1-kserve-6f556d9847-n8l8k"]
Sep 14 11:10:18.973658 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.973629    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:18.978825 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:18.978799    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:19.126476 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:19.126444    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:19.127262 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:19.127245    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:19.803858 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:19.803821    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4dd768c-253a-463f-bfb5-844bc5c5f243" path="/var/lib/kubelet/pods/d4dd768c-253a-463f-bfb5-844bc5c5f243/volumes"
Sep 14 11:10:24.387170 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.387126    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"]
Sep 14 11:10:24.387784 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.387764    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4dd768c-253a-463f-bfb5-844bc5c5f243" containerName="main"
Sep 14 11:10:24.387784 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.387786    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4dd768c-253a-463f-bfb5-844bc5c5f243" containerName="main"
Sep 14 11:10:24.387945 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.387914    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4dd768c-253a-463f-bfb5-844bc5c5f243" containerName="main"
Sep 14 11:10:24.392475 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.392452    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:24.396504 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.396485    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-22b014-serving-cert\""
Sep 14 11:10:24.423758 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.423734    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"]
Sep 14 11:10:24.461305 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.461273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:24.461449 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.461375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vld5j\" (UniqueName: \"kubernetes.io/projected/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-kube-api-access-vld5j\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:24.461449 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.461403    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:24.461535 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.461480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:24.562624 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.562576    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:24.562862 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.562666    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vld5j\" (UniqueName: \"kubernetes.io/projected/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-kube-api-access-vld5j\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:24.562862 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.562691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:24.562862 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.562766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:24.563062 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:24.562979    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-serving-cert: secret "maas-api-e2e-shared-b-w2-22b014-serving-cert" not found
Sep 14 11:10:24.563062 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:24.563048    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-maas-api-tls podName:5f21905f-da93-4a4b-91e4-cabfe6cb9bf0 nodeName:}" failed. No retries permitted until 2026-09-14 11:10:25.063025549 +0000 UTC m=+1329.747709915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-maas-api-tls") pod "maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp" (UID: "5f21905f-da93-4a4b-91e4-cabfe6cb9bf0") : secret "maas-api-e2e-shared-b-w2-22b014-serving-cert" not found
Sep 14 11:10:24.565800 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.565772    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:24.566218 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.566149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:24.582812 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:24.582791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vld5j\" (UniqueName: \"kubernetes.io/projected/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-kube-api-access-vld5j\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:25.068064 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:25.068026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:25.068224 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:25.068178    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-serving-cert: secret "maas-api-e2e-shared-b-w2-22b014-serving-cert" not found
Sep 14 11:10:25.068292 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:25.068242    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-maas-api-tls podName:5f21905f-da93-4a4b-91e4-cabfe6cb9bf0 nodeName:}" failed. No retries permitted until 2026-09-14 11:10:26.068226942 +0000 UTC m=+1330.752911309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-maas-api-tls") pod "maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp" (UID: "5f21905f-da93-4a4b-91e4-cabfe6cb9bf0") : secret "maas-api-e2e-shared-b-w2-22b014-serving-cert" not found
Sep 14 11:10:26.080970 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:26.080937    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:26.083295 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:26.083275    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:26.203115 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:26.203089    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:26.506373 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:26.506347    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"]
Sep 14 11:10:26.507172 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:10:26.507138    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f21905f_da93_4a4b_91e4_cabfe6cb9bf0.slice/crio-29eccbf60f8fdaa4d7044a04af42deb1a979ec39b82cb58753692a9be59fc639 WatchSource:0}: Error finding container 29eccbf60f8fdaa4d7044a04af42deb1a979ec39b82cb58753692a9be59fc639: Status 404 returned error can't find the container with id 29eccbf60f8fdaa4d7044a04af42deb1a979ec39b82cb58753692a9be59fc639
Sep 14 11:10:27.180246 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:27.180214    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp" event={"ID":"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0","Type":"ContainerStarted","Data":"ebfa76361b84166c34b3e113356164a364ca464e764133def3dd04fb66c18a7c"}
Sep 14 11:10:27.180684 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:27.180280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp" event={"ID":"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0","Type":"ContainerStarted","Data":"29eccbf60f8fdaa4d7044a04af42deb1a979ec39b82cb58753692a9be59fc639"}
Sep 14 11:10:27.180684 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:27.180385    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:29.956792 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:29.956738    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp" podStartSLOduration=5.528399783 podStartE2EDuration="5.956706432s" totalImagesPullingTime="428.306649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:27.250369043 +0000 UTC m=+1331.935053433" watchObservedRunningTime="2026-09-14 11:10:29.956706432 +0000 UTC m=+1334.641390820"
Sep 14 11:10:29.957199 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:29.957184    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"]
Sep 14 11:10:29.957382 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:29.957364    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7" podUID="3a1cf7e5-47bf-4845-9909-d0f45bb7d891" containerName="maas-api" containerID="cri-o://50bc05f22777702098903ed0fb8b29e1f88de304422d9253639cf71e73a59ece" gracePeriod=30
Sep 14 11:10:30.201326 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.201301    2561 generic.go:356] "Generic (PLEG): container finished" podID="3a1cf7e5-47bf-4845-9909-d0f45bb7d891" containerID="50bc05f22777702098903ed0fb8b29e1f88de304422d9253639cf71e73a59ece" exitCode=0
Sep 14 11:10:30.201423 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.201400    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7" event={"ID":"3a1cf7e5-47bf-4845-9909-d0f45bb7d891","Type":"ContainerDied","Data":"50bc05f22777702098903ed0fb8b29e1f88de304422d9253639cf71e73a59ece"}
Sep 14 11:10:30.222970 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.222947    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:10:30.326205 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.326176    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx699\" (UniqueName: \"kubernetes.io/projected/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-kube-api-access-kx699\") pod \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") "
Sep 14 11:10:30.326377 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.326243    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-openshift-service-ca\") pod \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") "
Sep 14 11:10:30.326377 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.326285    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-maas-api-tls\") pod \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") "
Sep 14 11:10:30.326377 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.326335    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-metrics-certs\") pod \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\" (UID: \"3a1cf7e5-47bf-4845-9909-d0f45bb7d891\") "
Sep 14 11:10:30.326611 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.326586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3a1cf7e5-47bf-4845-9909-d0f45bb7d891" (UID: "3a1cf7e5-47bf-4845-9909-d0f45bb7d891"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:10:30.328349 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.328315    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3a1cf7e5-47bf-4845-9909-d0f45bb7d891" (UID: "3a1cf7e5-47bf-4845-9909-d0f45bb7d891"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:10:30.328461 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.328353    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3a1cf7e5-47bf-4845-9909-d0f45bb7d891" (UID: "3a1cf7e5-47bf-4845-9909-d0f45bb7d891"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:10:30.328461 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.328368    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-kube-api-access-kx699" (OuterVolumeSpecName: "kube-api-access-kx699") pod "3a1cf7e5-47bf-4845-9909-d0f45bb7d891" (UID: "3a1cf7e5-47bf-4845-9909-d0f45bb7d891"). InnerVolumeSpecName "kube-api-access-kx699". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:10:30.434597 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.428541    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:30.434597 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.428581    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx699\" (UniqueName: \"kubernetes.io/projected/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-kube-api-access-kx699\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:30.434597 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.428596    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:30.434597 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:30.428610    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1cf7e5-47bf-4845-9909-d0f45bb7d891-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:31.208355 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:31.208317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7" event={"ID":"3a1cf7e5-47bf-4845-9909-d0f45bb7d891","Type":"ContainerDied","Data":"8dfcd975568f8304bd11d7da8240a983c42c60c1272f6fa0e908a39bc95c0c03"}
Sep 14 11:10:31.208355 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:31.208334    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"
Sep 14 11:10:31.209008 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:31.208379    2561 scope.go:117] "RemoveContainer" containerID="50bc05f22777702098903ed0fb8b29e1f88de304422d9253639cf71e73a59ece"
Sep 14 11:10:31.253962 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:31.253937    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"]
Sep 14 11:10:31.270926 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:31.270902    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0777c1-7589d79c8b-5m7b7"]
Sep 14 11:10:31.803396 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:31.803356    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a1cf7e5-47bf-4845-9909-d0f45bb7d891" path="/var/lib/kubelet/pods/3a1cf7e5-47bf-4845-9909-d0f45bb7d891/volumes"
Sep 14 11:10:33.193796 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:33.193763    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:10:37.741150 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:37.741119    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"]
Sep 14 11:10:37.741563 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:37.741313    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p" podUID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerName="istio-proxy" containerID="cri-o://cbed4184e7f96a5c0ee297ee4e447c18a613eb98d3f979909018287ebdcde9fd" gracePeriod=30
Sep 14 11:10:38.617379 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.617335    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"]
Sep 14 11:10:38.617811 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.617798    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a1cf7e5-47bf-4845-9909-d0f45bb7d891" containerName="maas-api"
Sep 14 11:10:38.617861 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.617812    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a1cf7e5-47bf-4845-9909-d0f45bb7d891" containerName="maas-api"
Sep 14 11:10:38.617897 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.617884    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a1cf7e5-47bf-4845-9909-d0f45bb7d891" containerName="maas-api"
Sep 14 11:10:38.622365 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.622349    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.625009 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.624989    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-fdbc00-openshift-default-dockercfg-qqwrd\""
Sep 14 11:10:38.640018 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.639986    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"]
Sep 14 11:10:38.710193 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.710163    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-podinfo\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.710344 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.710218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-token\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.710344 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.710240    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-envoy\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.710344 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.710298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-data\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.710344 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.710332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-workload-certs\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.710547 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.710355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f557605-f940-4ae6-9054-3dc0cee19d46-istiod-ca-cert\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.710547 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.710384    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-credential-socket\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.710547 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.710457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlz6j\" (UniqueName: \"kubernetes.io/projected/5f557605-f940-4ae6-9054-3dc0cee19d46-kube-api-access-dlz6j\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.710683 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.710545    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-workload-socket\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.811155 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.811124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-token\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.811155 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.811159    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-envoy\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.811688 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.811188    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-data\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.811688 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.811311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-workload-certs\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.811688 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.811362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f557605-f940-4ae6-9054-3dc0cee19d46-istiod-ca-cert\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.811688 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.811401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-credential-socket\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.811688 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.811433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlz6j\" (UniqueName: \"kubernetes.io/projected/5f557605-f940-4ae6-9054-3dc0cee19d46-kube-api-access-dlz6j\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.811688 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.811564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-workload-socket\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.811688 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.811616    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-data\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.811688 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.811673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-workload-certs\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.812179 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.811747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-credential-socket\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.812179 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.811679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-podinfo\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.812179 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.812141    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f557605-f940-4ae6-9054-3dc0cee19d46-istiod-ca-cert\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.812466 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.812260    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-workload-socket\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.813621 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.813596    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-envoy\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.813990 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.813973    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-podinfo\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.827090 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.827065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlz6j\" (UniqueName: \"kubernetes.io/projected/5f557605-f940-4ae6-9054-3dc0cee19d46-kube-api-access-dlz6j\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.830165 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.830141    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-token\") pod \"e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") " pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:38.934010 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:38.933914    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:39.225527 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:39.225500    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"]
Sep 14 11:10:39.226662 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:10:39.226636    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f557605_f940_4ae6_9054_3dc0cee19d46.slice/crio-3f0c2cc2cbcac5a65be1e3973f8fb96b69f6ba423ae22cf5577316d76c00babf WatchSource:0}: Error finding container 3f0c2cc2cbcac5a65be1e3973f8fb96b69f6ba423ae22cf5577316d76c00babf: Status 404 returned error can't find the container with id 3f0c2cc2cbcac5a65be1e3973f8fb96b69f6ba423ae22cf5577316d76c00babf
Sep 14 11:10:39.228835 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:39.228803    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:10:39.228898 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:39.228868    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:10:39.228941 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:39.228897    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:10:39.266163 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:39.266138    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" event={"ID":"5f557605-f940-4ae6-9054-3dc0cee19d46","Type":"ContainerStarted","Data":"3f0c2cc2cbcac5a65be1e3973f8fb96b69f6ba423ae22cf5577316d76c00babf"}
Sep 14 11:10:40.273607 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:40.273573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" event={"ID":"5f557605-f940-4ae6-9054-3dc0cee19d46","Type":"ContainerStarted","Data":"6bafa7e4d7ccb808fb9e5df059516243770b2ce3b4bed7b11c6d115ee265e90c"}
Sep 14 11:10:40.298430 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:40.298381    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" podStartSLOduration=2.298366721 podStartE2EDuration="2.298366721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:40.296329056 +0000 UTC m=+1344.981013444" watchObservedRunningTime="2026-09-14 11:10:40.298366721 +0000 UTC m=+1344.983051110"
Sep 14 11:10:40.586600 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:40.586522    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7842-311a41"]
Sep 14 11:10:40.590214 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:40.590195    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-311a41"
Sep 14 11:10:40.592886 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:40.592866    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-stw9c\""
Sep 14 11:10:40.602847 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:40.600089    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-311a41"]
Sep 14 11:10:40.630371 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:40.630345    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs672\" (UniqueName: \"kubernetes.io/projected/3ac2a0aa-88b5-4776-99b5-2609b5ce669f-kube-api-access-qs672\") pod \"test-curl-7842-311a41\" (UID: \"3ac2a0aa-88b5-4776-99b5-2609b5ce669f\") " pod="opendatahub/test-curl-7842-311a41"
Sep 14 11:10:40.731653 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:40.731630    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs672\" (UniqueName: \"kubernetes.io/projected/3ac2a0aa-88b5-4776-99b5-2609b5ce669f-kube-api-access-qs672\") pod \"test-curl-7842-311a41\" (UID: \"3ac2a0aa-88b5-4776-99b5-2609b5ce669f\") " pod="opendatahub/test-curl-7842-311a41"
Sep 14 11:10:40.743113 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:40.743086    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs672\" (UniqueName: \"kubernetes.io/projected/3ac2a0aa-88b5-4776-99b5-2609b5ce669f-kube-api-access-qs672\") pod \"test-curl-7842-311a41\" (UID: \"3ac2a0aa-88b5-4776-99b5-2609b5ce669f\") " pod="opendatahub/test-curl-7842-311a41"
Sep 14 11:10:40.901792 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:40.901764    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-311a41"
Sep 14 11:10:40.934236 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:40.934209    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:40.940040 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:40.939574    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:41.195372 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:41.195342    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-311a41"]
Sep 14 11:10:41.196427 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:10:41.196392    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ac2a0aa_88b5_4776_99b5_2609b5ce669f.slice/crio-d77f09889d8582445a74b63fdec246927a0e002b4d69bb6d68dee188f5630496 WatchSource:0}: Error finding container d77f09889d8582445a74b63fdec246927a0e002b4d69bb6d68dee188f5630496: Status 404 returned error can't find the container with id d77f09889d8582445a74b63fdec246927a0e002b4d69bb6d68dee188f5630496
Sep 14 11:10:41.281337 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:41.281307    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-311a41" event={"ID":"3ac2a0aa-88b5-4776-99b5-2609b5ce669f","Type":"ContainerStarted","Data":"73684771e089c55dae90131a56c99bff931f12cb1589ad819ee439a30490c221"}
Sep 14 11:10:41.281760 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:41.281354    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-311a41" event={"ID":"3ac2a0aa-88b5-4776-99b5-2609b5ce669f","Type":"ContainerStarted","Data":"d77f09889d8582445a74b63fdec246927a0e002b4d69bb6d68dee188f5630496"}
Sep 14 11:10:41.281760 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:41.281604    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:41.282322 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:41.282302    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:10:41.312152 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:41.312108    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7842-311a41" podStartSLOduration=1.312090667 podStartE2EDuration="1.312090667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:41.311644347 +0000 UTC m=+1345.996328737" watchObservedRunningTime="2026-09-14 11:10:41.312090667 +0000 UTC m=+1345.996775057"
Sep 14 11:10:41.915889 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:41.915849    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7842-311a41"]
Sep 14 11:10:41.919130 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:41.919100    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7842-311a41"]
Sep 14 11:10:42.026316 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:42.026284    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7842-a3392d"]
Sep 14 11:10:42.026702 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:42.026690    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac2a0aa-88b5-4776-99b5-2609b5ce669f" containerName="test-curl-7842-311a41"
Sep 14 11:10:42.026770 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:42.026704    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac2a0aa-88b5-4776-99b5-2609b5ce669f" containerName="test-curl-7842-311a41"
Sep 14 11:10:42.026817 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:42.026803    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac2a0aa-88b5-4776-99b5-2609b5ce669f" containerName="test-curl-7842-311a41"
Sep 14 11:10:42.030036 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:42.030015    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-a3392d"
Sep 14 11:10:42.038146 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:42.038123    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-a3392d"]
Sep 14 11:10:42.045346 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:42.045319    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k87rv\" (UniqueName: \"kubernetes.io/projected/bc1d7186-cba4-4078-a7f8-8e00cf13ef19-kube-api-access-k87rv\") pod \"test-curl-7842-a3392d\" (UID: \"bc1d7186-cba4-4078-a7f8-8e00cf13ef19\") " pod="opendatahub/test-curl-7842-a3392d"
Sep 14 11:10:42.146650 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:42.146614    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k87rv\" (UniqueName: \"kubernetes.io/projected/bc1d7186-cba4-4078-a7f8-8e00cf13ef19-kube-api-access-k87rv\") pod \"test-curl-7842-a3392d\" (UID: \"bc1d7186-cba4-4078-a7f8-8e00cf13ef19\") " pod="opendatahub/test-curl-7842-a3392d"
Sep 14 11:10:42.155233 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:42.155208    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k87rv\" (UniqueName: \"kubernetes.io/projected/bc1d7186-cba4-4078-a7f8-8e00cf13ef19-kube-api-access-k87rv\") pod \"test-curl-7842-a3392d\" (UID: \"bc1d7186-cba4-4078-a7f8-8e00cf13ef19\") " pod="opendatahub/test-curl-7842-a3392d"
Sep 14 11:10:42.287666 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:42.287584    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7842-311a41" container="test-curl-7842-311a41"
Sep 14 11:10:42.340552 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:42.340525    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-a3392d"
Sep 14 11:10:42.629020 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:42.628996    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-a3392d"]
Sep 14 11:10:42.630834 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:10:42.630805    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc1d7186_cba4_4078_a7f8_8e00cf13ef19.slice/crio-20fdcb20a14109300b14c342a8ab44a1cb68912a1e0ee253aec8e59855e67fc2 WatchSource:0}: Error finding container 20fdcb20a14109300b14c342a8ab44a1cb68912a1e0ee253aec8e59855e67fc2: Status 404 returned error can't find the container with id 20fdcb20a14109300b14c342a8ab44a1cb68912a1e0ee253aec8e59855e67fc2
Sep 14 11:10:42.982455 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:42.982431    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:10:43.055069 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055043    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-workload-certs\") pod \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") "
Sep 14 11:10:43.055069 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055082    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-token\") pod \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") "
Sep 14 11:10:43.055293 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055104    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istiod-ca-cert\") pod \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") "
Sep 14 11:10:43.055293 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055178    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-credential-socket\") pod \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") "
Sep 14 11:10:43.055293 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfbhv\" (UniqueName: \"kubernetes.io/projected/af96dc32-00b9-4d16-8303-3a4ddb5cb146-kube-api-access-nfbhv\") pod \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") "
Sep 14 11:10:43.055293 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055223    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-envoy\") pod \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") "
Sep 14 11:10:43.055293 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055239    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-podinfo\") pod \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") "
Sep 14 11:10:43.055293 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055259    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-workload-socket\") pod \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") "
Sep 14 11:10:43.055293 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055287    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-data\") pod \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\" (UID: \"af96dc32-00b9-4d16-8303-3a4ddb5cb146\") "
Sep 14 11:10:43.055670 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055306    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af96dc32-00b9-4d16-8303-3a4ddb5cb146" (UID: "af96dc32-00b9-4d16-8303-3a4ddb5cb146"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:43.055670 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055527    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af96dc32-00b9-4d16-8303-3a4ddb5cb146" (UID: "af96dc32-00b9-4d16-8303-3a4ddb5cb146"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:43.055670 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055599    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af96dc32-00b9-4d16-8303-3a4ddb5cb146" (UID: "af96dc32-00b9-4d16-8303-3a4ddb5cb146"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:10:43.055854 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055775    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-data" (OuterVolumeSpecName: "istio-data") pod "af96dc32-00b9-4d16-8303-3a4ddb5cb146" (UID: "af96dc32-00b9-4d16-8303-3a4ddb5cb146"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:43.055854 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055811    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af96dc32-00b9-4d16-8303-3a4ddb5cb146" (UID: "af96dc32-00b9-4d16-8303-3a4ddb5cb146"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:43.055964 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055876    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-credential-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:43.055964 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055890    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-data\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:43.055964 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055903    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-workload-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:43.055964 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.055916    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istiod-ca-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:43.057165 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.057140    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-token" (OuterVolumeSpecName: "istio-token") pod "af96dc32-00b9-4d16-8303-3a4ddb5cb146" (UID: "af96dc32-00b9-4d16-8303-3a4ddb5cb146"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:10:43.057405 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.057384    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af96dc32-00b9-4d16-8303-3a4ddb5cb146-kube-api-access-nfbhv" (OuterVolumeSpecName: "kube-api-access-nfbhv") pod "af96dc32-00b9-4d16-8303-3a4ddb5cb146" (UID: "af96dc32-00b9-4d16-8303-3a4ddb5cb146"). InnerVolumeSpecName "kube-api-access-nfbhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:10:43.057699 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.057677    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af96dc32-00b9-4d16-8303-3a4ddb5cb146" (UID: "af96dc32-00b9-4d16-8303-3a4ddb5cb146"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:43.057782 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.057684    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af96dc32-00b9-4d16-8303-3a4ddb5cb146" (UID: "af96dc32-00b9-4d16-8303-3a4ddb5cb146"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 11:10:43.157178 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.157110    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:43.157178 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.157132    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfbhv\" (UniqueName: \"kubernetes.io/projected/af96dc32-00b9-4d16-8303-3a4ddb5cb146-kube-api-access-nfbhv\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:43.157178 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.157142    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-envoy\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:43.157178 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.157150    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af96dc32-00b9-4d16-8303-3a4ddb5cb146-istio-podinfo\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:43.157178 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.157158    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af96dc32-00b9-4d16-8303-3a4ddb5cb146-workload-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:43.296025 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.295987    2561 generic.go:356] "Generic (PLEG): container finished" podID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerID="cbed4184e7f96a5c0ee297ee4e447c18a613eb98d3f979909018287ebdcde9fd" exitCode=0
Sep 14 11:10:43.296379 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.296047    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"
Sep 14 11:10:43.296379 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.296075    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p" event={"ID":"af96dc32-00b9-4d16-8303-3a4ddb5cb146","Type":"ContainerDied","Data":"cbed4184e7f96a5c0ee297ee4e447c18a613eb98d3f979909018287ebdcde9fd"}
Sep 14 11:10:43.296379 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.296132    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p" event={"ID":"af96dc32-00b9-4d16-8303-3a4ddb5cb146","Type":"ContainerDied","Data":"6dd7a9264e506212090e818b8e3697d9576f4576d6abfb97a3a770fb883a8b22"}
Sep 14 11:10:43.296379 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.296160    2561 scope.go:117] "RemoveContainer" containerID="cbed4184e7f96a5c0ee297ee4e447c18a613eb98d3f979909018287ebdcde9fd"
Sep 14 11:10:43.297965 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.297944    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-a3392d" event={"ID":"bc1d7186-cba4-4078-a7f8-8e00cf13ef19","Type":"ContainerStarted","Data":"8e242562b910cd52d6d2ce3897b6ca328b0cb6f94c76af84ac56cc7ca9c6a348"}
Sep 14 11:10:43.298063 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.297980    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-a3392d" event={"ID":"bc1d7186-cba4-4078-a7f8-8e00cf13ef19","Type":"ContainerStarted","Data":"20fdcb20a14109300b14c342a8ab44a1cb68912a1e0ee253aec8e59855e67fc2"}
Sep 14 11:10:43.298241 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.298213    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7842-311a41" podUID="3ac2a0aa-88b5-4776-99b5-2609b5ce669f" containerName="test-curl-7842-311a41" containerID="cri-o://73684771e089c55dae90131a56c99bff931f12cb1589ad819ee439a30490c221" gracePeriod=2
Sep 14 11:10:43.306136 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.306119    2561 scope.go:117] "RemoveContainer" containerID="2de697a2e8020e08f8f6577bc77d62c1822a1c9991eed06584796ff02e709566"
Sep 14 11:10:43.315358 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.315332    2561 scope.go:117] "RemoveContainer" containerID="cbed4184e7f96a5c0ee297ee4e447c18a613eb98d3f979909018287ebdcde9fd"
Sep 14 11:10:43.315628 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.315608    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:10:43.315702 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:43.315637    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbed4184e7f96a5c0ee297ee4e447c18a613eb98d3f979909018287ebdcde9fd\": container with ID starting with cbed4184e7f96a5c0ee297ee4e447c18a613eb98d3f979909018287ebdcde9fd not found: ID does not exist" containerID="cbed4184e7f96a5c0ee297ee4e447c18a613eb98d3f979909018287ebdcde9fd"
Sep 14 11:10:43.315702 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.315668    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbed4184e7f96a5c0ee297ee4e447c18a613eb98d3f979909018287ebdcde9fd"} err="failed to get container status \"cbed4184e7f96a5c0ee297ee4e447c18a613eb98d3f979909018287ebdcde9fd\": rpc error: code = NotFound desc = could not find container \"cbed4184e7f96a5c0ee297ee4e447c18a613eb98d3f979909018287ebdcde9fd\": container with ID starting with cbed4184e7f96a5c0ee297ee4e447c18a613eb98d3f979909018287ebdcde9fd not found: ID does not exist"
Sep 14 11:10:43.315702 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.315690    2561 scope.go:117] "RemoveContainer" containerID="2de697a2e8020e08f8f6577bc77d62c1822a1c9991eed06584796ff02e709566"
Sep 14 11:10:43.316025 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:43.315999    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de697a2e8020e08f8f6577bc77d62c1822a1c9991eed06584796ff02e709566\": container with ID starting with 2de697a2e8020e08f8f6577bc77d62c1822a1c9991eed06584796ff02e709566 not found: ID does not exist" containerID="2de697a2e8020e08f8f6577bc77d62c1822a1c9991eed06584796ff02e709566"
Sep 14 11:10:43.316076 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.316035    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de697a2e8020e08f8f6577bc77d62c1822a1c9991eed06584796ff02e709566"} err="failed to get container status \"2de697a2e8020e08f8f6577bc77d62c1822a1c9991eed06584796ff02e709566\": rpc error: code = NotFound desc = could not find container \"2de697a2e8020e08f8f6577bc77d62c1822a1c9991eed06584796ff02e709566\": container with ID starting with 2de697a2e8020e08f8f6577bc77d62c1822a1c9991eed06584796ff02e709566 not found: ID does not exist"
Sep 14 11:10:43.329251 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:43.329220    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7842-311a41\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="3ac2a0aa-88b5-4776-99b5-2609b5ce669f" pod="opendatahub/test-curl-7842-311a41"
Sep 14 11:10:43.329366 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.329254    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7842-a3392d" podStartSLOduration=1.329238918 podStartE2EDuration="1.329238918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:43.326949005 +0000 UTC m=+1348.011633406" watchObservedRunningTime="2026-09-14 11:10:43.329238918 +0000 UTC m=+1348.013923309"
Sep 14 11:10:43.345576 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.345545    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"]
Sep 14 11:10:43.353269 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:43.353240    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7842-311a41\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="3ac2a0aa-88b5-4776-99b5-2609b5ce669f" pod="opendatahub/test-curl-7842-311a41"
Sep 14 11:10:43.355059 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.355035    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0777c1-openshift-default-8d776c9d-5pc5p"]
Sep 14 11:10:43.805295 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.805218    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" path="/var/lib/kubelet/pods/af96dc32-00b9-4d16-8303-3a4ddb5cb146/volumes"
Sep 14 11:10:43.977286 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.977249    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7842-a3392d"]
Sep 14 11:10:43.984036 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:43.983999    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7842-a3392d"]
Sep 14 11:10:44.100766 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.100736    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7842-b246fc"]
Sep 14 11:10:44.101834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.101813    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerName="istio-proxy"
Sep 14 11:10:44.101913 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.101838    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerName="istio-proxy"
Sep 14 11:10:44.101913 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.101882    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1d7186-cba4-4078-a7f8-8e00cf13ef19" containerName="test-curl-7842-a3392d"
Sep 14 11:10:44.101913 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.101892    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1d7186-cba4-4078-a7f8-8e00cf13ef19" containerName="test-curl-7842-a3392d"
Sep 14 11:10:44.102130 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.102117    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1d7186-cba4-4078-a7f8-8e00cf13ef19" containerName="test-curl-7842-a3392d"
Sep 14 11:10:44.102168 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.102139    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerName="istio-proxy"
Sep 14 11:10:44.108286 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.108263    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-b246fc"
Sep 14 11:10:44.114004 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.113981    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-b246fc"]
Sep 14 11:10:44.168529 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.168501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6zks\" (UniqueName: \"kubernetes.io/projected/d45faa6a-7167-4472-9a65-e2eee4f2f8cb-kube-api-access-r6zks\") pod \"test-curl-7842-b246fc\" (UID: \"d45faa6a-7167-4472-9a65-e2eee4f2f8cb\") " pod="opendatahub/test-curl-7842-b246fc"
Sep 14 11:10:44.270179 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.270133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zks\" (UniqueName: \"kubernetes.io/projected/d45faa6a-7167-4472-9a65-e2eee4f2f8cb-kube-api-access-r6zks\") pod \"test-curl-7842-b246fc\" (UID: \"d45faa6a-7167-4472-9a65-e2eee4f2f8cb\") " pod="opendatahub/test-curl-7842-b246fc"
Sep 14 11:10:44.281502 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.281477    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6zks\" (UniqueName: \"kubernetes.io/projected/d45faa6a-7167-4472-9a65-e2eee4f2f8cb-kube-api-access-r6zks\") pod \"test-curl-7842-b246fc\" (UID: \"d45faa6a-7167-4472-9a65-e2eee4f2f8cb\") " pod="opendatahub/test-curl-7842-b246fc"
Sep 14 11:10:44.306577 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:44.306554    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7842-a3392d" container="test-curl-7842-a3392d"
Sep 14 11:10:44.420976 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.420943    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-b246fc"
Sep 14 11:10:44.705480 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:44.705451    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-b246fc"]
Sep 14 11:10:44.706270 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:10:44.706241    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd45faa6a_7167_4472_9a65_e2eee4f2f8cb.slice/crio-c1d653bcf60c2d2dd81e630599443c0597e785fe3d9b010f8ff2d42769cb42c3 WatchSource:0}: Error finding container c1d653bcf60c2d2dd81e630599443c0597e785fe3d9b010f8ff2d42769cb42c3: Status 404 returned error can't find the container with id c1d653bcf60c2d2dd81e630599443c0597e785fe3d9b010f8ff2d42769cb42c3
Sep 14 11:10:45.314131 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:45.314102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-b246fc" event={"ID":"d45faa6a-7167-4472-9a65-e2eee4f2f8cb","Type":"ContainerStarted","Data":"d65be7269aad535e0c36c6e8d23dc19a6df5bb2855d2e6cf3a4abf2e449c6670"}
Sep 14 11:10:45.314421 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:45.314146    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-b246fc" event={"ID":"d45faa6a-7167-4472-9a65-e2eee4f2f8cb","Type":"ContainerStarted","Data":"c1d653bcf60c2d2dd81e630599443c0597e785fe3d9b010f8ff2d42769cb42c3"}
Sep 14 11:10:45.314421 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:45.314234    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7842-a3392d" podUID="bc1d7186-cba4-4078-a7f8-8e00cf13ef19" containerName="test-curl-7842-a3392d" containerID="cri-o://8e242562b910cd52d6d2ce3897b6ca328b0cb6f94c76af84ac56cc7ca9c6a348" gracePeriod=2
Sep 14 11:10:45.330417 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:45.330375    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7842-b246fc" podStartSLOduration=1.330360543 podStartE2EDuration="1.330360543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:45.328499126 +0000 UTC m=+1350.013183515" watchObservedRunningTime="2026-09-14 11:10:45.330360543 +0000 UTC m=+1350.015044932"
Sep 14 11:10:45.330745 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:45.330708    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7842-a3392d\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="bc1d7186-cba4-4078-a7f8-8e00cf13ef19" pod="opendatahub/test-curl-7842-a3392d"
Sep 14 11:10:45.460088 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:45.460052    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-311a41"
Sep 14 11:10:45.462917 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:45.462876    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7842-a3392d\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="bc1d7186-cba4-4078-a7f8-8e00cf13ef19" pod="opendatahub/test-curl-7842-a3392d"
Sep 14 11:10:45.582909 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:45.582841    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs672\" (UniqueName: \"kubernetes.io/projected/3ac2a0aa-88b5-4776-99b5-2609b5ce669f-kube-api-access-qs672\") pod \"3ac2a0aa-88b5-4776-99b5-2609b5ce669f\" (UID: \"3ac2a0aa-88b5-4776-99b5-2609b5ce669f\") "
Sep 14 11:10:45.584748 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:45.584728    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac2a0aa-88b5-4776-99b5-2609b5ce669f-kube-api-access-qs672" (OuterVolumeSpecName: "kube-api-access-qs672") pod "3ac2a0aa-88b5-4776-99b5-2609b5ce669f" (UID: "3ac2a0aa-88b5-4776-99b5-2609b5ce669f"). InnerVolumeSpecName "kube-api-access-qs672". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:10:45.683999 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:45.683970    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs672\" (UniqueName: \"kubernetes.io/projected/3ac2a0aa-88b5-4776-99b5-2609b5ce669f-kube-api-access-qs672\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:45.802240 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:45.802194    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7842-a3392d\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="bc1d7186-cba4-4078-a7f8-8e00cf13ef19" pod="opendatahub/test-curl-7842-a3392d"
Sep 14 11:10:45.803492 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:45.803471    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ac2a0aa-88b5-4776-99b5-2609b5ce669f" path="/var/lib/kubelet/pods/3ac2a0aa-88b5-4776-99b5-2609b5ce669f/volumes"
Sep 14 11:10:45.908795 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:45.908768    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7842-b246fc"]
Sep 14 11:10:45.913999 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:45.913973    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7842-b246fc"]
Sep 14 11:10:46.026543 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.026511    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7842-4790a5"]
Sep 14 11:10:46.026965 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.026951    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerName="istio-proxy"
Sep 14 11:10:46.027019 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.026967    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerName="istio-proxy"
Sep 14 11:10:46.027019 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.026989    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d45faa6a-7167-4472-9a65-e2eee4f2f8cb" containerName="test-curl-7842-b246fc"
Sep 14 11:10:46.027019 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.026995    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d45faa6a-7167-4472-9a65-e2eee4f2f8cb" containerName="test-curl-7842-b246fc"
Sep 14 11:10:46.027118 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.027058    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="af96dc32-00b9-4d16-8303-3a4ddb5cb146" containerName="istio-proxy"
Sep 14 11:10:46.027118 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.027066    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d45faa6a-7167-4472-9a65-e2eee4f2f8cb" containerName="test-curl-7842-b246fc"
Sep 14 11:10:46.030287 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.030270    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-4790a5"
Sep 14 11:10:46.037982 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.037891    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-4790a5"]
Sep 14 11:10:46.088254 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.088210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsjg\" (UniqueName: \"kubernetes.io/projected/eee7df73-1b32-4bb0-a950-587d04f1c21b-kube-api-access-hwsjg\") pod \"test-curl-7842-4790a5\" (UID: \"eee7df73-1b32-4bb0-a950-587d04f1c21b\") " pod="opendatahub/test-curl-7842-4790a5"
Sep 14 11:10:46.141350 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.141321    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"]
Sep 14 11:10:46.145630 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.144988    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:46.165399 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.161160    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-fdbc00-serving-cert\""
Sep 14 11:10:46.168255 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.166003    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"]
Sep 14 11:10:46.188946 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.188915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2tr6\" (UniqueName: \"kubernetes.io/projected/88d570b9-8384-4305-95b1-a2326de4100a-kube-api-access-l2tr6\") pod \"maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:46.189083 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.189062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d570b9-8384-4305-95b1-a2326de4100a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:46.189148 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.189103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:46.189219 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.189148    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:46.189308 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.189293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsjg\" (UniqueName: \"kubernetes.io/projected/eee7df73-1b32-4bb0-a950-587d04f1c21b-kube-api-access-hwsjg\") pod \"test-curl-7842-4790a5\" (UID: \"eee7df73-1b32-4bb0-a950-587d04f1c21b\") " pod="opendatahub/test-curl-7842-4790a5"
Sep 14 11:10:46.199655 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.199590    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsjg\" (UniqueName: \"kubernetes.io/projected/eee7df73-1b32-4bb0-a950-587d04f1c21b-kube-api-access-hwsjg\") pod \"test-curl-7842-4790a5\" (UID: \"eee7df73-1b32-4bb0-a950-587d04f1c21b\") " pod="opendatahub/test-curl-7842-4790a5"
Sep 14 11:10:46.290338 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.290300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d570b9-8384-4305-95b1-a2326de4100a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:46.290533 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.290367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:46.290533 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.290404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:46.290533 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.290504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2tr6\" (UniqueName: \"kubernetes.io/projected/88d570b9-8384-4305-95b1-a2326de4100a-kube-api-access-l2tr6\") pod \"maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:46.290760 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:46.290530    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-serving-cert: secret "maas-api-e2e-shared-a-w3-fdbc00-serving-cert" not found
Sep 14 11:10:46.290760 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:46.290601    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-maas-api-tls podName:88d570b9-8384-4305-95b1-a2326de4100a nodeName:}" failed. No retries permitted until 2026-09-14 11:10:46.790582182 +0000 UTC m=+1351.475266562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-maas-api-tls") pod "maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv" (UID: "88d570b9-8384-4305-95b1-a2326de4100a") : secret "maas-api-e2e-shared-a-w3-fdbc00-serving-cert" not found
Sep 14 11:10:46.291118 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.291097    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d570b9-8384-4305-95b1-a2326de4100a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:46.292923 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.292906    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:46.300297 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.300273    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2tr6\" (UniqueName: \"kubernetes.io/projected/88d570b9-8384-4305-95b1-a2326de4100a-kube-api-access-l2tr6\") pod \"maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:46.324314 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.324282    2561 generic.go:356] "Generic (PLEG): container finished" podID="3ac2a0aa-88b5-4776-99b5-2609b5ce669f" containerID="73684771e089c55dae90131a56c99bff931f12cb1589ad819ee439a30490c221" exitCode=137
Sep 14 11:10:46.324701 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.324322    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-311a41"
Sep 14 11:10:46.324701 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.324367    2561 scope.go:117] "RemoveContainer" containerID="73684771e089c55dae90131a56c99bff931f12cb1589ad819ee439a30490c221"
Sep 14 11:10:46.324701 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:46.324684    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7842-b246fc" container="test-curl-7842-b246fc"
Sep 14 11:10:46.333397 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.333375    2561 scope.go:117] "RemoveContainer" containerID="73684771e089c55dae90131a56c99bff931f12cb1589ad819ee439a30490c221"
Sep 14 11:10:46.333676 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:46.333657    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73684771e089c55dae90131a56c99bff931f12cb1589ad819ee439a30490c221\": container with ID starting with 73684771e089c55dae90131a56c99bff931f12cb1589ad819ee439a30490c221 not found: ID does not exist" containerID="73684771e089c55dae90131a56c99bff931f12cb1589ad819ee439a30490c221"
Sep 14 11:10:46.333764 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.333685    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73684771e089c55dae90131a56c99bff931f12cb1589ad819ee439a30490c221"} err="failed to get container status \"73684771e089c55dae90131a56c99bff931f12cb1589ad819ee439a30490c221\": rpc error: code = NotFound desc = could not find container \"73684771e089c55dae90131a56c99bff931f12cb1589ad819ee439a30490c221\": container with ID starting with 73684771e089c55dae90131a56c99bff931f12cb1589ad819ee439a30490c221 not found: ID does not exist"
Sep 14 11:10:46.341225 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.341206    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-4790a5"
Sep 14 11:10:46.641864 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.641779    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-4790a5"]
Sep 14 11:10:46.643318 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:10:46.643287    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeee7df73_1b32_4bb0_a950_587d04f1c21b.slice/crio-63cd749b5d07f5fddad22001db3b126256e636a7c36cb69db15afa31e4cf9cc8 WatchSource:0}: Error finding container 63cd749b5d07f5fddad22001db3b126256e636a7c36cb69db15afa31e4cf9cc8: Status 404 returned error can't find the container with id 63cd749b5d07f5fddad22001db3b126256e636a7c36cb69db15afa31e4cf9cc8
Sep 14 11:10:46.796556 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.796518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:46.798965 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:46.798945    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:47.086000 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:47.085908    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:47.336463 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:47.336385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-4790a5" event={"ID":"eee7df73-1b32-4bb0-a950-587d04f1c21b","Type":"ContainerStarted","Data":"10d9f10ef5336b5555ab33272b495c5758f1ffe4566e1cc3c32e427cbec5ee7f"}
Sep 14 11:10:47.336463 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:47.336439    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-4790a5" event={"ID":"eee7df73-1b32-4bb0-a950-587d04f1c21b","Type":"ContainerStarted","Data":"63cd749b5d07f5fddad22001db3b126256e636a7c36cb69db15afa31e4cf9cc8"}
Sep 14 11:10:47.337594 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:47.337567    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7842-b246fc" podUID="d45faa6a-7167-4472-9a65-e2eee4f2f8cb" containerName="test-curl-7842-b246fc" containerID="cri-o://d65be7269aad535e0c36c6e8d23dc19a6df5bb2855d2e6cf3a4abf2e449c6670" gracePeriod=2
Sep 14 11:10:47.373922 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:47.373870    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7842-b246fc\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="d45faa6a-7167-4472-9a65-e2eee4f2f8cb" pod="opendatahub/test-curl-7842-b246fc"
Sep 14 11:10:47.374854 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:47.374330    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7842-4790a5" podStartSLOduration=1.374312212 podStartE2EDuration="1.374312212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:47.369990419 +0000 UTC m=+1352.054674808" watchObservedRunningTime="2026-09-14 11:10:47.374312212 +0000 UTC m=+1352.058996607"
Sep 14 11:10:47.390829 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:47.390796    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"]
Sep 14 11:10:47.392737 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:10:47.392650    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d570b9_8384_4305_95b1_a2326de4100a.slice/crio-f84fd11c63ecdfb942a1d2df68337e4becc448a2fb6f92b248b05a087e5a2c75 WatchSource:0}: Error finding container f84fd11c63ecdfb942a1d2df68337e4becc448a2fb6f92b248b05a087e5a2c75: Status 404 returned error can't find the container with id f84fd11c63ecdfb942a1d2df68337e4becc448a2fb6f92b248b05a087e5a2c75
Sep 14 11:10:47.503290 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:47.503261    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-a3392d"
Sep 14 11:10:47.505942 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:47.505897    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7842-b246fc\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="d45faa6a-7167-4472-9a65-e2eee4f2f8cb" pod="opendatahub/test-curl-7842-b246fc"
Sep 14 11:10:47.604665 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:47.604641    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k87rv\" (UniqueName: \"kubernetes.io/projected/bc1d7186-cba4-4078-a7f8-8e00cf13ef19-kube-api-access-k87rv\") pod \"bc1d7186-cba4-4078-a7f8-8e00cf13ef19\" (UID: \"bc1d7186-cba4-4078-a7f8-8e00cf13ef19\") "
Sep 14 11:10:47.606693 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:47.606661    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1d7186-cba4-4078-a7f8-8e00cf13ef19-kube-api-access-k87rv" (OuterVolumeSpecName: "kube-api-access-k87rv") pod "bc1d7186-cba4-4078-a7f8-8e00cf13ef19" (UID: "bc1d7186-cba4-4078-a7f8-8e00cf13ef19"). InnerVolumeSpecName "kube-api-access-k87rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:10:47.706043 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:47.706016    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k87rv\" (UniqueName: \"kubernetes.io/projected/bc1d7186-cba4-4078-a7f8-8e00cf13ef19-kube-api-access-k87rv\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:47.807004 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:47.806899    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1d7186-cba4-4078-a7f8-8e00cf13ef19" path="/var/lib/kubelet/pods/bc1d7186-cba4-4078-a7f8-8e00cf13ef19/volumes"
Sep 14 11:10:48.029685 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.029602    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7842-4790a5"]
Sep 14 11:10:48.036680 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.036650    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7842-4790a5"]
Sep 14 11:10:48.346005 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.345956    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv" event={"ID":"88d570b9-8384-4305-95b1-a2326de4100a","Type":"ContainerStarted","Data":"a7d9e93c3bc37fdae139ad359e190fd6847b54d36ee8051376d9e43e1db5f574"}
Sep 14 11:10:48.346448 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.346034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv" event={"ID":"88d570b9-8384-4305-95b1-a2326de4100a","Type":"ContainerStarted","Data":"f84fd11c63ecdfb942a1d2df68337e4becc448a2fb6f92b248b05a087e5a2c75"}
Sep 14 11:10:48.346448 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.346089    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:48.347848 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.347820    2561 generic.go:356] "Generic (PLEG): container finished" podID="bc1d7186-cba4-4078-a7f8-8e00cf13ef19" containerID="8e242562b910cd52d6d2ce3897b6ca328b0cb6f94c76af84ac56cc7ca9c6a348" exitCode=137
Sep 14 11:10:48.347945 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.347874    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-a3392d"
Sep 14 11:10:48.348027 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.347930    2561 scope.go:117] "RemoveContainer" containerID="8e242562b910cd52d6d2ce3897b6ca328b0cb6f94c76af84ac56cc7ca9c6a348"
Sep 14 11:10:48.348119 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:48.348099    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7842-4790a5" container="test-curl-7842-4790a5"
Sep 14 11:10:48.365974 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.364620    2561 scope.go:117] "RemoveContainer" containerID="8e242562b910cd52d6d2ce3897b6ca328b0cb6f94c76af84ac56cc7ca9c6a348"
Sep 14 11:10:48.365974 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:48.365015    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e242562b910cd52d6d2ce3897b6ca328b0cb6f94c76af84ac56cc7ca9c6a348\": container with ID starting with 8e242562b910cd52d6d2ce3897b6ca328b0cb6f94c76af84ac56cc7ca9c6a348 not found: ID does not exist" containerID="8e242562b910cd52d6d2ce3897b6ca328b0cb6f94c76af84ac56cc7ca9c6a348"
Sep 14 11:10:48.365974 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.365050    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e242562b910cd52d6d2ce3897b6ca328b0cb6f94c76af84ac56cc7ca9c6a348"} err="failed to get container status \"8e242562b910cd52d6d2ce3897b6ca328b0cb6f94c76af84ac56cc7ca9c6a348\": rpc error: code = NotFound desc = could not find container \"8e242562b910cd52d6d2ce3897b6ca328b0cb6f94c76af84ac56cc7ca9c6a348\": container with ID starting with 8e242562b910cd52d6d2ce3897b6ca328b0cb6f94c76af84ac56cc7ca9c6a348 not found: ID does not exist"
Sep 14 11:10:48.377232 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.377179    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv" podStartSLOduration=2.027845236 podStartE2EDuration="2.377162722s" totalImagesPullingTime="349.317486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:48.375356686 +0000 UTC m=+1353.060041087" watchObservedRunningTime="2026-09-14 11:10:48.377162722 +0000 UTC m=+1353.061847124"
Sep 14 11:10:48.963735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.962739    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"]
Sep 14 11:10:48.963735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.963400    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee7df73-1b32-4bb0-a950-587d04f1c21b" containerName="test-curl-7842-4790a5"
Sep 14 11:10:48.963735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.963421    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee7df73-1b32-4bb0-a950-587d04f1c21b" containerName="test-curl-7842-4790a5"
Sep 14 11:10:48.963735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.963543    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee7df73-1b32-4bb0-a950-587d04f1c21b" containerName="test-curl-7842-4790a5"
Sep 14 11:10:48.979409 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.977942    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:48.980604 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.980570    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"]
Sep 14 11:10:48.981327 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:48.981092    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e6e0b4a3-openshift-default-dockercfg-sp7kk\""
Sep 14 11:10:49.018004 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.017977    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-credential-socket\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.018115 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.018077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-podinfo\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.018161 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.018118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp65d\" (UniqueName: \"kubernetes.io/projected/b24897dc-1da7-489f-ac43-b7515a99f29f-kube-api-access-cp65d\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.018161 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.018149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-workload-certs\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.018239 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.018170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-envoy\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.018239 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.018187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-token\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.018239 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.018210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-workload-socket\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.018389 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.018243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24897dc-1da7-489f-ac43-b7515a99f29f-istiod-ca-cert\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.018389 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.018295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-data\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.118823 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.118794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-workload-certs\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.118977 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.118831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-envoy\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.118977 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.118850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-token\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.118977 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.118872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-workload-socket\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.118977 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.118918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24897dc-1da7-489f-ac43-b7515a99f29f-istiod-ca-cert\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.118977 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.118935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-data\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.119153 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.118984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-credential-socket\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.119153 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.119031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-podinfo\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.119153 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.119067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp65d\" (UniqueName: \"kubernetes.io/projected/b24897dc-1da7-489f-ac43-b7515a99f29f-kube-api-access-cp65d\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.119254 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.119225    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-workload-certs\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.119481 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.119457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-data\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.119626 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.119602    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-workload-socket\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.119755 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.119735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-credential-socket\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.119961 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.119935    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24897dc-1da7-489f-ac43-b7515a99f29f-istiod-ca-cert\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.121321 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.121300    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-envoy\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.121886 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.121866    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-podinfo\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.128973 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.128953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-token\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.129421 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.129403    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp65d\" (UniqueName: \"kubernetes.io/projected/b24897dc-1da7-489f-ac43-b7515a99f29f-kube-api-access-cp65d\") pod \"e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") " pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.294199 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.294104    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:49.387438 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.387180    2561 generic.go:356] "Generic (PLEG): container finished" podID="d45faa6a-7167-4472-9a65-e2eee4f2f8cb" containerID="d65be7269aad535e0c36c6e8d23dc19a6df5bb2855d2e6cf3a4abf2e449c6670" exitCode=137
Sep 14 11:10:49.390118 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.389974    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7842-4790a5" podUID="eee7df73-1b32-4bb0-a950-587d04f1c21b" containerName="test-curl-7842-4790a5" containerID="cri-o://10d9f10ef5336b5555ab33272b495c5758f1ffe4566e1cc3c32e427cbec5ee7f" gracePeriod=2
Sep 14 11:10:49.393510 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:49.393478    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7842-4790a5\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="eee7df73-1b32-4bb0-a950-587d04f1c21b" pod="opendatahub/test-curl-7842-4790a5"
Sep 14 11:10:49.535873 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.535846    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-b246fc"
Sep 14 11:10:49.538324 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:49.538290    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7842-4790a5\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="eee7df73-1b32-4bb0-a950-587d04f1c21b" pod="opendatahub/test-curl-7842-4790a5"
Sep 14 11:10:49.597922 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.597894    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"]
Sep 14 11:10:49.599120 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:10:49.599093    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24897dc_1da7_489f_ac43_b7515a99f29f.slice/crio-8fcfd47ed43bd56440b5c90f4a8a8ad11b5b8d6df6ac69b2ebf77e9cf5192d43 WatchSource:0}: Error finding container 8fcfd47ed43bd56440b5c90f4a8a8ad11b5b8d6df6ac69b2ebf77e9cf5192d43: Status 404 returned error can't find the container with id 8fcfd47ed43bd56440b5c90f4a8a8ad11b5b8d6df6ac69b2ebf77e9cf5192d43
Sep 14 11:10:49.601575 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.601545    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:10:49.601652 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.601609    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:10:49.601652 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.601635    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:10:49.623902 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.623879    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6zks\" (UniqueName: \"kubernetes.io/projected/d45faa6a-7167-4472-9a65-e2eee4f2f8cb-kube-api-access-r6zks\") pod \"d45faa6a-7167-4472-9a65-e2eee4f2f8cb\" (UID: \"d45faa6a-7167-4472-9a65-e2eee4f2f8cb\") "
Sep 14 11:10:49.625685 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.625662    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45faa6a-7167-4472-9a65-e2eee4f2f8cb-kube-api-access-r6zks" (OuterVolumeSpecName: "kube-api-access-r6zks") pod "d45faa6a-7167-4472-9a65-e2eee4f2f8cb" (UID: "d45faa6a-7167-4472-9a65-e2eee4f2f8cb"). InnerVolumeSpecName "kube-api-access-r6zks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:10:49.725306 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.725267    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6zks\" (UniqueName: \"kubernetes.io/projected/d45faa6a-7167-4472-9a65-e2eee4f2f8cb-kube-api-access-r6zks\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:49.805212 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:49.805179    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d45faa6a-7167-4472-9a65-e2eee4f2f8cb" path="/var/lib/kubelet/pods/d45faa6a-7167-4472-9a65-e2eee4f2f8cb/volumes"
Sep 14 11:10:50.398303 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:50.398265    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8" event={"ID":"b24897dc-1da7-489f-ac43-b7515a99f29f","Type":"ContainerStarted","Data":"b6eac89989875cb84b34f6f6de64fa54d95096a4840f1ffb94f2c3c8f1acfdae"}
Sep 14 11:10:50.398737 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:50.398362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8" event={"ID":"b24897dc-1da7-489f-ac43-b7515a99f29f","Type":"ContainerStarted","Data":"8fcfd47ed43bd56440b5c90f4a8a8ad11b5b8d6df6ac69b2ebf77e9cf5192d43"}
Sep 14 11:10:50.399546 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:50.399519    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-b246fc"
Sep 14 11:10:50.399546 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:50.399535    2561 scope.go:117] "RemoveContainer" containerID="d65be7269aad535e0c36c6e8d23dc19a6df5bb2855d2e6cf3a4abf2e449c6670"
Sep 14 11:10:50.422441 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:50.422402    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8" podStartSLOduration=2.422388457 podStartE2EDuration="2.422388457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:50.421089239 +0000 UTC m=+1355.105773640" watchObservedRunningTime="2026-09-14 11:10:50.422388457 +0000 UTC m=+1355.107072845"
Sep 14 11:10:51.294581 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:51.294546    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:51.299599 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:51.299574    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:51.407608 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:51.407582    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:51.408548 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:51.408532    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:10:51.598752 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:51.598731    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-4790a5"
Sep 14 11:10:51.645336 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:51.645307    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwsjg\" (UniqueName: \"kubernetes.io/projected/eee7df73-1b32-4bb0-a950-587d04f1c21b-kube-api-access-hwsjg\") pod \"eee7df73-1b32-4bb0-a950-587d04f1c21b\" (UID: \"eee7df73-1b32-4bb0-a950-587d04f1c21b\") "
Sep 14 11:10:51.647486 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:51.647459    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee7df73-1b32-4bb0-a950-587d04f1c21b-kube-api-access-hwsjg" (OuterVolumeSpecName: "kube-api-access-hwsjg") pod "eee7df73-1b32-4bb0-a950-587d04f1c21b" (UID: "eee7df73-1b32-4bb0-a950-587d04f1c21b"). InnerVolumeSpecName "kube-api-access-hwsjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:10:51.749022 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:51.747249    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwsjg\" (UniqueName: \"kubernetes.io/projected/eee7df73-1b32-4bb0-a950-587d04f1c21b-kube-api-access-hwsjg\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:10:51.803837 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:51.803802    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eee7df73-1b32-4bb0-a950-587d04f1c21b" path="/var/lib/kubelet/pods/eee7df73-1b32-4bb0-a950-587d04f1c21b/volumes"
Sep 14 11:10:52.415930 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:52.415899    2561 generic.go:356] "Generic (PLEG): container finished" podID="eee7df73-1b32-4bb0-a950-587d04f1c21b" containerID="10d9f10ef5336b5555ab33272b495c5758f1ffe4566e1cc3c32e427cbec5ee7f" exitCode=137
Sep 14 11:10:52.416423 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:52.415950    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-4790a5"
Sep 14 11:10:52.416423 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:52.416012    2561 scope.go:117] "RemoveContainer" containerID="10d9f10ef5336b5555ab33272b495c5758f1ffe4566e1cc3c32e427cbec5ee7f"
Sep 14 11:10:52.426980 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:52.426954    2561 scope.go:117] "RemoveContainer" containerID="10d9f10ef5336b5555ab33272b495c5758f1ffe4566e1cc3c32e427cbec5ee7f"
Sep 14 11:10:52.427289 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:52.427265    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d9f10ef5336b5555ab33272b495c5758f1ffe4566e1cc3c32e427cbec5ee7f\": container with ID starting with 10d9f10ef5336b5555ab33272b495c5758f1ffe4566e1cc3c32e427cbec5ee7f not found: ID does not exist" containerID="10d9f10ef5336b5555ab33272b495c5758f1ffe4566e1cc3c32e427cbec5ee7f"
Sep 14 11:10:52.427390 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:52.427297    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d9f10ef5336b5555ab33272b495c5758f1ffe4566e1cc3c32e427cbec5ee7f"} err="failed to get container status \"10d9f10ef5336b5555ab33272b495c5758f1ffe4566e1cc3c32e427cbec5ee7f\": rpc error: code = NotFound desc = could not find container \"10d9f10ef5336b5555ab33272b495c5758f1ffe4566e1cc3c32e427cbec5ee7f\": container with ID starting with 10d9f10ef5336b5555ab33272b495c5758f1ffe4566e1cc3c32e427cbec5ee7f not found: ID does not exist"
Sep 14 11:10:54.395924 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:54.395897    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:10:55.437551 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:55.437516    2561 generic.go:356] "Generic (PLEG): container finished" podID="9e419258-d263-4df1-aaf0-422eace67cc9" containerID="71db1e8a767f07cfb39c07d2dd500d7a61a3b6deb07b70eddec204826147ad90" exitCode=0
Sep 14 11:10:55.438080 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:55.437591    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv" event={"ID":"9e419258-d263-4df1-aaf0-422eace67cc9","Type":"ContainerDied","Data":"71db1e8a767f07cfb39c07d2dd500d7a61a3b6deb07b70eddec204826147ad90"}
Sep 14 11:10:55.438147 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:55.438132    2561 scope.go:117] "RemoveContainer" containerID="71db1e8a767f07cfb39c07d2dd500d7a61a3b6deb07b70eddec204826147ad90"
Sep 14 11:10:55.439245 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:55.439215    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:10:55.439359 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:55.439283    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:10:55.439359 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:55.439315    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:10:56.446766 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:56.446701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv" event={"ID":"9e419258-d263-4df1-aaf0-422eace67cc9","Type":"ContainerStarted","Data":"02d9283172b7b2ba2a7ab2d8304070e06ad3231d00c0191462787726cca96a55"}
Sep 14 11:10:56.974171 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:56.974137    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:10:56.974360 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:56.974328    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 14 11:10:56.974427 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:56.974402    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv" podUID="9e419258-d263-4df1-aaf0-422eace67cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 14 11:10:57.130725 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.130567    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"]
Sep 14 11:10:57.134778 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.134755    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.138395 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.138211    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e6e0b4a3-serving-cert\""
Sep 14 11:10:57.179567 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.162308    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"]
Sep 14 11:10:57.202313 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.202280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-maas-api-tls\") pod \"maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.202474 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.202327    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d064825-4fd6-4d93-96fd-099007232ab5-openshift-service-ca\") pod \"maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.202474 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.202351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd6kh\" (UniqueName: \"kubernetes.io/projected/3d064825-4fd6-4d93-96fd-099007232ab5-kube-api-access-cd6kh\") pod \"maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.202474 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.202404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-metrics-certs\") pod \"maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.303437 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.303354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-maas-api-tls\") pod \"maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.303437 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.303404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d064825-4fd6-4d93-96fd-099007232ab5-openshift-service-ca\") pod \"maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.303437 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.303437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd6kh\" (UniqueName: \"kubernetes.io/projected/3d064825-4fd6-4d93-96fd-099007232ab5-kube-api-access-cd6kh\") pod \"maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.303667 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.303520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-metrics-certs\") pod \"maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.303870 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:57.303841    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-serving-cert: secret "maas-api-e2e-mt-e6e0b4a3-serving-cert" not found
Sep 14 11:10:57.303992 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:10:57.303915    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-maas-api-tls podName:3d064825-4fd6-4d93-96fd-099007232ab5 nodeName:}" failed. No retries permitted until 2026-09-14 11:10:57.803895937 +0000 UTC m=+1362.488580308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-maas-api-tls") pod "maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj" (UID: "3d064825-4fd6-4d93-96fd-099007232ab5") : secret "maas-api-e2e-mt-e6e0b4a3-serving-cert" not found
Sep 14 11:10:57.304264 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.304235    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d064825-4fd6-4d93-96fd-099007232ab5-openshift-service-ca\") pod \"maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.306092 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.306070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-metrics-certs\") pod \"maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.316798 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.316776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd6kh\" (UniqueName: \"kubernetes.io/projected/3d064825-4fd6-4d93-96fd-099007232ab5-kube-api-access-cd6kh\") pod \"maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.809116 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.809082    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-maas-api-tls\") pod \"maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.811462 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.811443    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-maas-api-tls\") pod \"maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:57.976736 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.976281    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 14 11:10:57.976736 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:57.976344    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv" podUID="9e419258-d263-4df1-aaf0-422eace67cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 14 11:10:58.072789 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:58.071925    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:58.359373 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:58.359340    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"]
Sep 14 11:10:58.471960 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:58.471919    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj" event={"ID":"3d064825-4fd6-4d93-96fd-099007232ab5","Type":"ContainerStarted","Data":"4e6b9cb46b3bff5be5d83d5e5fe1b7187635e58580ae617f3712050df9caa9db"}
Sep 14 11:10:58.974136 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:58.974051    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 14 11:10:58.974136 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:58.974120    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv" podUID="9e419258-d263-4df1-aaf0-422eace67cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 14 11:10:59.490737 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:59.490537    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj" event={"ID":"3d064825-4fd6-4d93-96fd-099007232ab5","Type":"ContainerStarted","Data":"9a936e49571172dd79747717400e15cb2726ed7003a36c57c711947e28bfe21c"}
Sep 14 11:10:59.490737 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:59.490663    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:10:59.519751 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:59.519677    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj" podStartSLOduration=2.176271506 podStartE2EDuration="2.519655599s" totalImagesPullingTime="343.384093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:59.518823308 +0000 UTC m=+1364.203507700" watchObservedRunningTime="2026-09-14 11:10:59.519655599 +0000 UTC m=+1364.204339986"
Sep 14 11:10:59.974031 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:59.973999    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 14 11:10:59.974262 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:10:59.974241    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv" podUID="9e419258-d263-4df1-aaf0-422eace67cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 14 11:11:00.973874 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:00.973842    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 14 11:11:00.974110 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:00.974087    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv" podUID="9e419258-d263-4df1-aaf0-422eace67cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 14 11:11:01.977346 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:01.977315    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:11:01.977934 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:01.977637    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:11:01.978556 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:01.978536    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:11:05.503397 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:05.503364    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:11:11.545459 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.545421    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"]
Sep 14 11:11:11.552132 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.552106    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:11.557669 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.557638    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-0fb86d-serving-cert\""
Sep 14 11:11:11.572379 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.572354    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"]
Sep 14 11:11:11.656275 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.656237    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:11.656457 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.656315    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f445db-c180-4d42-838e-11d7fe0fc0ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:11.656457 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.656367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:11.656457 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.656410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mrv2\" (UniqueName: \"kubernetes.io/projected/e6f445db-c180-4d42-838e-11d7fe0fc0ad-kube-api-access-4mrv2\") pod \"maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:11.757629 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.757598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:11.757853 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.757672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f445db-c180-4d42-838e-11d7fe0fc0ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:11.757853 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.757704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:11.757853 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.757743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mrv2\" (UniqueName: \"kubernetes.io/projected/e6f445db-c180-4d42-838e-11d7fe0fc0ad-kube-api-access-4mrv2\") pod \"maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:11.757853 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:11:11.757808    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-serving-cert: secret "maas-api-e2e-shared-b-w3-0fb86d-serving-cert" not found
Sep 14 11:11:11.758092 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:11:11.757906    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-maas-api-tls podName:e6f445db-c180-4d42-838e-11d7fe0fc0ad nodeName:}" failed. No retries permitted until 2026-09-14 11:11:12.257884 +0000 UTC m=+1376.942568392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-maas-api-tls") pod "maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j" (UID: "e6f445db-c180-4d42-838e-11d7fe0fc0ad") : secret "maas-api-e2e-shared-b-w3-0fb86d-serving-cert" not found
Sep 14 11:11:11.758463 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.758441    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f445db-c180-4d42-838e-11d7fe0fc0ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:11.760224 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.760198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:11.766494 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:11.766471    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mrv2\" (UniqueName: \"kubernetes.io/projected/e6f445db-c180-4d42-838e-11d7fe0fc0ad-kube-api-access-4mrv2\") pod \"maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:12.262562 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:12.262516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:12.264884 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:12.264862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:12.324850 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:12.324817    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 11:11:12.324986 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:12.324876    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 11:11:12.468756 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:12.468699    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:12.755745 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:12.755700    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"]
Sep 14 11:11:12.756376 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:11:12.756352    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f445db_c180_4d42_838e_11d7fe0fc0ad.slice/crio-0e06646b555bb2b2d3a935dc6d55be9f8f96c91a739e8ca5d4d827066605b3c4 WatchSource:0}: Error finding container 0e06646b555bb2b2d3a935dc6d55be9f8f96c91a739e8ca5d4d827066605b3c4: Status 404 returned error can't find the container with id 0e06646b555bb2b2d3a935dc6d55be9f8f96c91a739e8ca5d4d827066605b3c4
Sep 14 11:11:13.583488 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:13.583401    2561 generic.go:356] "Generic (PLEG): container finished" podID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerID="6bafa7e4d7ccb808fb9e5df059516243770b2ce3b4bed7b11c6d115ee265e90c" exitCode=0
Sep 14 11:11:13.583643 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:13.583479    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" event={"ID":"5f557605-f940-4ae6-9054-3dc0cee19d46","Type":"ContainerDied","Data":"6bafa7e4d7ccb808fb9e5df059516243770b2ce3b4bed7b11c6d115ee265e90c"}
Sep 14 11:11:13.583998 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:13.583976    2561 scope.go:117] "RemoveContainer" containerID="6bafa7e4d7ccb808fb9e5df059516243770b2ce3b4bed7b11c6d115ee265e90c"
Sep 14 11:11:13.585203 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:13.585179    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j" event={"ID":"e6f445db-c180-4d42-838e-11d7fe0fc0ad","Type":"ContainerStarted","Data":"d89454fbfe9cf589d597c7ad25c20fe444271d41265286182e1b7f0095b3efed"}
Sep 14 11:11:13.585314 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:13.585222    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:11:13.585314 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:13.585239    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:13.585314 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:13.585267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j" event={"ID":"e6f445db-c180-4d42-838e-11d7fe0fc0ad","Type":"ContainerStarted","Data":"0e06646b555bb2b2d3a935dc6d55be9f8f96c91a739e8ca5d4d827066605b3c4"}
Sep 14 11:11:13.585314 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:13.585287    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:11:13.585457 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:13.585324    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:11:13.626794 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:13.626749    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j" podStartSLOduration=2.180492147 podStartE2EDuration="2.626704075s" totalImagesPullingTime="446.211928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:11:13.62453267 +0000 UTC m=+1378.309217060" watchObservedRunningTime="2026-09-14 11:11:13.626704075 +0000 UTC m=+1378.311388464"
Sep 14 11:11:14.117314 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:14.117281    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:11:14.594167 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:14.594072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" event={"ID":"5f557605-f940-4ae6-9054-3dc0cee19d46","Type":"ContainerStarted","Data":"a80414b796e0fcdcba2c1bb3b7536122a3c05827f3b88f7a97dcffc7c47e318f"}
Sep 14 11:11:14.934658 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:14.934621    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:11:14.934861 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:14.934809    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 14 11:11:14.934911 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:14.934860    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 14 11:11:15.934418 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:15.934382    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 14 11:11:15.934877 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:15.934440    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 14 11:11:16.934675 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:16.934626    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 14 11:11:16.935101 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:16.934681    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 14 11:11:17.934872 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:17.934838    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 14 11:11:17.935253 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:17.934894    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 14 11:11:18.934544 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:18.934501    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 14 11:11:18.934762 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:18.934566    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 14 11:11:19.603407 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:19.603380    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:11:19.935133 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:19.935090    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 14 11:11:19.935351 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:19.935157    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 14 11:11:20.938377 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:20.938346    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:11:20.938849 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:20.938825    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:11:20.939454 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:20.939434    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:11:25.675558 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:25.675526    2561 generic.go:356] "Generic (PLEG): container finished" podID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerID="b6eac89989875cb84b34f6f6de64fa54d95096a4840f1ffb94f2c3c8f1acfdae" exitCode=0
Sep 14 11:11:25.675996 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:25.675566    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8" event={"ID":"b24897dc-1da7-489f-ac43-b7515a99f29f","Type":"ContainerDied","Data":"b6eac89989875cb84b34f6f6de64fa54d95096a4840f1ffb94f2c3c8f1acfdae"}
Sep 14 11:11:25.676052 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:25.675994    2561 scope.go:117] "RemoveContainer" containerID="b6eac89989875cb84b34f6f6de64fa54d95096a4840f1ffb94f2c3c8f1acfdae"
Sep 14 11:11:25.677148 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:25.677120    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:11:25.677232 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:25.677185    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:11:25.677232 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:25.677215    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:11:26.683964 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:26.683930    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8" event={"ID":"b24897dc-1da7-489f-ac43-b7515a99f29f","Type":"ContainerStarted","Data":"cc4667ac8212e916fc5a017af7716332bbdbc75195088fe4a8b61c71216afbb2"}
Sep 14 11:11:27.295006 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:27.294967    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:11:27.295212 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:27.295188    2561 patch_prober.go:28] interesting pod/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 14 11:11:27.295268 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:27.295250    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8" podUID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 14 11:11:28.295210 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:28.295169    2561 patch_prober.go:28] interesting pod/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 14 11:11:28.295615 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:28.295228    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8" podUID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 14 11:11:29.295022 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:29.294984    2561 patch_prober.go:28] interesting pod/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 14 11:11:29.295201 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:29.295054    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8" podUID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 14 11:11:30.295312 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:30.295278    2561 patch_prober.go:28] interesting pod/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 14 11:11:30.295708 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:30.295334    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8" podUID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 14 11:11:31.295038 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:31.295008    2561 patch_prober.go:28] interesting pod/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 14 11:11:31.295205 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:31.295073    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8" podUID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 14 11:11:32.298828 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:32.298799    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:11:32.299269 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:32.299257    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:11:32.299876 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:32.299856    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:11:46.453470 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:46.453425    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:11:57.352358 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.352324    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"]
Sep 14 11:11:57.356172 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.356155    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.359335 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.359314    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e6e0b4a3\"/\"openshift-service-ca.crt\""
Sep 14 11:11:57.360432 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.360411    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e6e0b4a3\"/\"kube-root-ca.crt\""
Sep 14 11:11:57.360540 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.360489    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e6e0b4a3\"/\"e2e-lifecycle-model-e6e0b4a3-kserve-self-signed-certs\""
Sep 14 11:11:57.360540 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.360492    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e6e0b4a3\"/\"default-dockercfg-lwpf9\""
Sep 14 11:11:57.367052 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.367033    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"]
Sep 14 11:11:57.502094 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.502061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-home\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.502269 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.502102    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjvt9\" (UniqueName: \"kubernetes.io/projected/c970ae4e-5d23-4004-a4a0-fa1156695de5-kube-api-access-rjvt9\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.502269 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.502130    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-model-cache\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.502269 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.502232    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-tmp-dir\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.502378 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.502272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-dshm\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.502378 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.502362    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c970ae4e-5d23-4004-a4a0-fa1156695de5-tls-certs\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.603419 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.603328    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-dshm\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.603584 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.603422    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c970ae4e-5d23-4004-a4a0-fa1156695de5-tls-certs\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.603584 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.603475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-home\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.603584 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.603515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvt9\" (UniqueName: \"kubernetes.io/projected/c970ae4e-5d23-4004-a4a0-fa1156695de5-kube-api-access-rjvt9\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.603584 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.603559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-model-cache\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.603849 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.603641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-tmp-dir\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.603982 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.603948    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-home\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.603982 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.603962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-model-cache\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.604158 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.604045    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-tmp-dir\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.605758 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.605737    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-dshm\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.605904 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.605887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c970ae4e-5d23-4004-a4a0-fa1156695de5-tls-certs\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.612991 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.612968    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjvt9\" (UniqueName: \"kubernetes.io/projected/c970ae4e-5d23-4004-a4a0-fa1156695de5-kube-api-access-rjvt9\") pod \"e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") " pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.666029 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.666004    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:11:57.951235 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:57.951203    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"]
Sep 14 11:11:57.952515 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:11:57.952487    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc970ae4e_5d23_4004_a4a0_fa1156695de5.slice/crio-9f3a0244a59da4d5df1c9b7212059c56c47d1156391c15749a84b7315074f2d1 WatchSource:0}: Error finding container 9f3a0244a59da4d5df1c9b7212059c56c47d1156391c15749a84b7315074f2d1: Status 404 returned error can't find the container with id 9f3a0244a59da4d5df1c9b7212059c56c47d1156391c15749a84b7315074f2d1
Sep 14 11:11:58.886334 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:58.886297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b" event={"ID":"c970ae4e-5d23-4004-a4a0-fa1156695de5","Type":"ContainerStarted","Data":"f20e1fbd35bc1d513ca55f94166153f7704608792c2972a88715505ceddcc72d"}
Sep 14 11:11:58.886334 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:58.886341    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b" event={"ID":"c970ae4e-5d23-4004-a4a0-fa1156695de5","Type":"ContainerStarted","Data":"9f3a0244a59da4d5df1c9b7212059c56c47d1156391c15749a84b7315074f2d1"}
Sep 14 11:11:58.917351 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:11:58.917297    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b" podStartSLOduration=1.917282092 podStartE2EDuration="1.917282092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:11:58.914607157 +0000 UTC m=+1423.599291559" watchObservedRunningTime="2026-09-14 11:11:58.917282092 +0000 UTC m=+1423.601966480"
Sep 14 11:12:07.667060 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:12:07.667021    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:12:07.667412 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:12:07.667078    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:12:07.679165 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:12:07.679141    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:12:07.958987 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:12:07.958899    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:12:15.424224 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:12:15.424190    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:12:52.526742 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:12:52.526695    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:13:14.424970 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.424940    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"]
Sep 14 11:13:14.425507 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.425160    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj" podUID="3d064825-4fd6-4d93-96fd-099007232ab5" containerName="maas-api" containerID="cri-o://9a936e49571172dd79747717400e15cb2726ed7003a36c57c711947e28bfe21c" gracePeriod=30
Sep 14 11:13:14.702735 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.702594    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:13:14.868960 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.868927    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-metrics-certs\") pod \"3d064825-4fd6-4d93-96fd-099007232ab5\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") "
Sep 14 11:13:14.869155 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.868970    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-maas-api-tls\") pod \"3d064825-4fd6-4d93-96fd-099007232ab5\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") "
Sep 14 11:13:14.869155 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.869029    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd6kh\" (UniqueName: \"kubernetes.io/projected/3d064825-4fd6-4d93-96fd-099007232ab5-kube-api-access-cd6kh\") pod \"3d064825-4fd6-4d93-96fd-099007232ab5\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") "
Sep 14 11:13:14.869155 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.869078    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d064825-4fd6-4d93-96fd-099007232ab5-openshift-service-ca\") pod \"3d064825-4fd6-4d93-96fd-099007232ab5\" (UID: \"3d064825-4fd6-4d93-96fd-099007232ab5\") "
Sep 14 11:13:14.869510 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.869475    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d064825-4fd6-4d93-96fd-099007232ab5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d064825-4fd6-4d93-96fd-099007232ab5" (UID: "3d064825-4fd6-4d93-96fd-099007232ab5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:13:14.869673 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.869642    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d064825-4fd6-4d93-96fd-099007232ab5-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:13:14.871129 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.871104    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d064825-4fd6-4d93-96fd-099007232ab5-kube-api-access-cd6kh" (OuterVolumeSpecName: "kube-api-access-cd6kh") pod "3d064825-4fd6-4d93-96fd-099007232ab5" (UID: "3d064825-4fd6-4d93-96fd-099007232ab5"). InnerVolumeSpecName "kube-api-access-cd6kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:13:14.871551 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.871529    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d064825-4fd6-4d93-96fd-099007232ab5" (UID: "3d064825-4fd6-4d93-96fd-099007232ab5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:13:14.871629 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.871564    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3d064825-4fd6-4d93-96fd-099007232ab5" (UID: "3d064825-4fd6-4d93-96fd-099007232ab5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:13:14.971223 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.971131    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:13:14.971223 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.971159    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d064825-4fd6-4d93-96fd-099007232ab5-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:13:14.971223 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:14.971168    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd6kh\" (UniqueName: \"kubernetes.io/projected/3d064825-4fd6-4d93-96fd-099007232ab5-kube-api-access-cd6kh\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:13:15.367047 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:15.367014    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d064825-4fd6-4d93-96fd-099007232ab5" containerID="9a936e49571172dd79747717400e15cb2726ed7003a36c57c711947e28bfe21c" exitCode=0
Sep 14 11:13:15.367217 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:15.367059    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"
Sep 14 11:13:15.367217 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:15.367067    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj" event={"ID":"3d064825-4fd6-4d93-96fd-099007232ab5","Type":"ContainerDied","Data":"9a936e49571172dd79747717400e15cb2726ed7003a36c57c711947e28bfe21c"}
Sep 14 11:13:15.367217 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:15.367115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj" event={"ID":"3d064825-4fd6-4d93-96fd-099007232ab5","Type":"ContainerDied","Data":"4e6b9cb46b3bff5be5d83d5e5fe1b7187635e58580ae617f3712050df9caa9db"}
Sep 14 11:13:15.367217 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:15.367138    2561 scope.go:117] "RemoveContainer" containerID="9a936e49571172dd79747717400e15cb2726ed7003a36c57c711947e28bfe21c"
Sep 14 11:13:15.377263 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:15.377244    2561 scope.go:117] "RemoveContainer" containerID="9a936e49571172dd79747717400e15cb2726ed7003a36c57c711947e28bfe21c"
Sep 14 11:13:15.377535 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:13:15.377518    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a936e49571172dd79747717400e15cb2726ed7003a36c57c711947e28bfe21c\": container with ID starting with 9a936e49571172dd79747717400e15cb2726ed7003a36c57c711947e28bfe21c not found: ID does not exist" containerID="9a936e49571172dd79747717400e15cb2726ed7003a36c57c711947e28bfe21c"
Sep 14 11:13:15.377594 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:15.377542    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a936e49571172dd79747717400e15cb2726ed7003a36c57c711947e28bfe21c"} err="failed to get container status \"9a936e49571172dd79747717400e15cb2726ed7003a36c57c711947e28bfe21c\": rpc error: code = NotFound desc = could not find container \"9a936e49571172dd79747717400e15cb2726ed7003a36c57c711947e28bfe21c\": container with ID starting with 9a936e49571172dd79747717400e15cb2726ed7003a36c57c711947e28bfe21c not found: ID does not exist"
Sep 14 11:13:15.409137 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:15.409112    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"]
Sep 14 11:13:15.417055 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:15.417035    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6e0b4a3-7ddc6647bb-p2fsj"]
Sep 14 11:13:15.805122 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:15.805048    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d064825-4fd6-4d93-96fd-099007232ab5" path="/var/lib/kubelet/pods/3d064825-4fd6-4d93-96fd-099007232ab5/volumes"
Sep 14 11:13:15.925933 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:15.925907    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 11:13:15.928524 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:15.928499    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 11:13:22.863900 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:22.863864    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"]
Sep 14 11:13:24.654137 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:24.654100    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:13:37.883205 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:37.883162    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b" podUID="c970ae4e-5d23-4004-a4a0-fa1156695de5" containerName="main" containerID="cri-o://f20e1fbd35bc1d513ca55f94166153f7704608792c2972a88715505ceddcc72d" gracePeriod=45
Sep 14 11:13:37.944021 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:37.943984    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b" podUID="c970ae4e-5d23-4004-a4a0-fa1156695de5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 14 11:13:38.138187 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.138135    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:13:38.194532 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.194504    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-model-cache\") pod \"c970ae4e-5d23-4004-a4a0-fa1156695de5\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") "
Sep 14 11:13:38.194740 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.194541    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-dshm\") pod \"c970ae4e-5d23-4004-a4a0-fa1156695de5\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") "
Sep 14 11:13:38.194740 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.194588    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c970ae4e-5d23-4004-a4a0-fa1156695de5-tls-certs\") pod \"c970ae4e-5d23-4004-a4a0-fa1156695de5\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") "
Sep 14 11:13:38.194740 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.194623    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-tmp-dir\") pod \"c970ae4e-5d23-4004-a4a0-fa1156695de5\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") "
Sep 14 11:13:38.194740 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.194651    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjvt9\" (UniqueName: \"kubernetes.io/projected/c970ae4e-5d23-4004-a4a0-fa1156695de5-kube-api-access-rjvt9\") pod \"c970ae4e-5d23-4004-a4a0-fa1156695de5\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") "
Sep 14 11:13:38.194740 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.194702    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-home\") pod \"c970ae4e-5d23-4004-a4a0-fa1156695de5\" (UID: \"c970ae4e-5d23-4004-a4a0-fa1156695de5\") "
Sep 14 11:13:38.195032 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.194802    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-model-cache" (OuterVolumeSpecName: "model-cache") pod "c970ae4e-5d23-4004-a4a0-fa1156695de5" (UID: "c970ae4e-5d23-4004-a4a0-fa1156695de5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:13:38.195032 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.194962    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c970ae4e-5d23-4004-a4a0-fa1156695de5" (UID: "c970ae4e-5d23-4004-a4a0-fa1156695de5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:13:38.195150 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.195040    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-home" (OuterVolumeSpecName: "home") pod "c970ae4e-5d23-4004-a4a0-fa1156695de5" (UID: "c970ae4e-5d23-4004-a4a0-fa1156695de5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:13:38.195223 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.195203    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-model-cache\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:13:38.195285 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.195229    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-tmp-dir\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:13:38.195285 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.195243    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-home\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:13:38.196952 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.196927    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-dshm" (OuterVolumeSpecName: "dshm") pod "c970ae4e-5d23-4004-a4a0-fa1156695de5" (UID: "c970ae4e-5d23-4004-a4a0-fa1156695de5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:13:38.197073 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.196962    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c970ae4e-5d23-4004-a4a0-fa1156695de5-kube-api-access-rjvt9" (OuterVolumeSpecName: "kube-api-access-rjvt9") pod "c970ae4e-5d23-4004-a4a0-fa1156695de5" (UID: "c970ae4e-5d23-4004-a4a0-fa1156695de5"). InnerVolumeSpecName "kube-api-access-rjvt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:13:38.197073 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.196998    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c970ae4e-5d23-4004-a4a0-fa1156695de5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c970ae4e-5d23-4004-a4a0-fa1156695de5" (UID: "c970ae4e-5d23-4004-a4a0-fa1156695de5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:13:38.296191 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.296159    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjvt9\" (UniqueName: \"kubernetes.io/projected/c970ae4e-5d23-4004-a4a0-fa1156695de5-kube-api-access-rjvt9\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:13:38.296191 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.296184    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c970ae4e-5d23-4004-a4a0-fa1156695de5-dshm\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:13:38.296191 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.296195    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c970ae4e-5d23-4004-a4a0-fa1156695de5-tls-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:13:38.510557 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.510476    2561 generic.go:356] "Generic (PLEG): container finished" podID="c970ae4e-5d23-4004-a4a0-fa1156695de5" containerID="f20e1fbd35bc1d513ca55f94166153f7704608792c2972a88715505ceddcc72d" exitCode=0
Sep 14 11:13:38.510557 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.510517    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"
Sep 14 11:13:38.510758 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.510566    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b" event={"ID":"c970ae4e-5d23-4004-a4a0-fa1156695de5","Type":"ContainerDied","Data":"f20e1fbd35bc1d513ca55f94166153f7704608792c2972a88715505ceddcc72d"}
Sep 14 11:13:38.510758 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.510609    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b" event={"ID":"c970ae4e-5d23-4004-a4a0-fa1156695de5","Type":"ContainerDied","Data":"9f3a0244a59da4d5df1c9b7212059c56c47d1156391c15749a84b7315074f2d1"}
Sep 14 11:13:38.510758 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.510631    2561 scope.go:117] "RemoveContainer" containerID="f20e1fbd35bc1d513ca55f94166153f7704608792c2972a88715505ceddcc72d"
Sep 14 11:13:38.520701 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.520680    2561 scope.go:117] "RemoveContainer" containerID="f20e1fbd35bc1d513ca55f94166153f7704608792c2972a88715505ceddcc72d"
Sep 14 11:13:38.521014 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:13:38.520996    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20e1fbd35bc1d513ca55f94166153f7704608792c2972a88715505ceddcc72d\": container with ID starting with f20e1fbd35bc1d513ca55f94166153f7704608792c2972a88715505ceddcc72d not found: ID does not exist" containerID="f20e1fbd35bc1d513ca55f94166153f7704608792c2972a88715505ceddcc72d"
Sep 14 11:13:38.521072 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.521022    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20e1fbd35bc1d513ca55f94166153f7704608792c2972a88715505ceddcc72d"} err="failed to get container status \"f20e1fbd35bc1d513ca55f94166153f7704608792c2972a88715505ceddcc72d\": rpc error: code = NotFound desc = could not find container \"f20e1fbd35bc1d513ca55f94166153f7704608792c2972a88715505ceddcc72d\": container with ID starting with f20e1fbd35bc1d513ca55f94166153f7704608792c2972a88715505ceddcc72d not found: ID does not exist"
Sep 14 11:13:38.534323 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.534299    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"]
Sep 14 11:13:38.538254 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:38.538224    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e6e0b4a3/e2e-lifecycle-model-e6e0b4a3-kserve-685c495779-zt97b"]
Sep 14 11:13:39.804243 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:39.804212    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c970ae4e-5d23-4004-a4a0-fa1156695de5" path="/var/lib/kubelet/pods/c970ae4e-5d23-4004-a4a0-fa1156695de5/volumes"
Sep 14 11:13:45.852327 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.852296    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"]
Sep 14 11:13:45.852797 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.852763    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d064825-4fd6-4d93-96fd-099007232ab5" containerName="maas-api"
Sep 14 11:13:45.852797 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.852776    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d064825-4fd6-4d93-96fd-099007232ab5" containerName="maas-api"
Sep 14 11:13:45.852797 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.852797    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c970ae4e-5d23-4004-a4a0-fa1156695de5" containerName="main"
Sep 14 11:13:45.852906 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.852805    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c970ae4e-5d23-4004-a4a0-fa1156695de5" containerName="main"
Sep 14 11:13:45.852906 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.852870    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d064825-4fd6-4d93-96fd-099007232ab5" containerName="maas-api"
Sep 14 11:13:45.852906 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.852881    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c970ae4e-5d23-4004-a4a0-fa1156695de5" containerName="main"
Sep 14 11:13:45.857299 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.857275    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:45.861785 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.861754    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7984b1\"/\"kube-root-ca.crt\""
Sep 14 11:13:45.861785 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.861772    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7984b1\"/\"openshift-service-ca.crt\""
Sep 14 11:13:45.861953 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.861764    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7984b1\"/\"embedding-w0-7984b1-kserve-self-signed-certs\""
Sep 14 11:13:45.861953 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.861764    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7984b1\"/\"default-dockercfg-ff6n2\""
Sep 14 11:13:45.875843 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.875822    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"]
Sep 14 11:13:45.973924 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.973892    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-tls-certs\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:45.974083 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.973956    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-home\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:45.974083 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.974011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-tmp-dir\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:45.974083 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.974035    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zczvn\" (UniqueName: \"kubernetes.io/projected/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-kube-api-access-zczvn\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:45.974083 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.974065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-dshm\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:45.974245 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:45.974128    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-model-cache\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.075479 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.075441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-dshm\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.075677 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.075502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-model-cache\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.075677 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.075591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-tls-certs\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.075677 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.075653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-home\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.075879 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.075702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-tmp-dir\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.075879 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.075744    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zczvn\" (UniqueName: \"kubernetes.io/projected/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-kube-api-access-zczvn\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.076222 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.076196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-model-cache\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.076354 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.076243    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-home\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.076354 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.076312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-tmp-dir\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.077956 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.077930    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-dshm\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.078186 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.078168    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-tls-certs\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.085131 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.085107    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zczvn\" (UniqueName: \"kubernetes.io/projected/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-kube-api-access-zczvn\") pod \"embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") " pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.166689 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.166660    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:46.465265 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.465227    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"]
Sep 14 11:13:46.568736 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.568679    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v" event={"ID":"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab","Type":"ContainerStarted","Data":"65269aae114e88dca920cdd17907d834b9dc8d5d5f23dcd6565a71ffc5db4f6e"}
Sep 14 11:13:46.568909 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.568789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v" event={"ID":"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab","Type":"ContainerStarted","Data":"f4992a1ca74854e768f4e5677a57fd2624dccff7995933400124e0d3598277c5"}
Sep 14 11:13:46.592118 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:46.592067    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v" podStartSLOduration=1.592051306 podStartE2EDuration="1.592051306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:13:46.590609122 +0000 UTC m=+1531.275293514" watchObservedRunningTime="2026-09-14 11:13:46.592051306 +0000 UTC m=+1531.276735694"
Sep 14 11:13:56.167605 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:56.167566    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:56.168039 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:56.167643    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:56.180036 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:56.180013    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:13:56.641656 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:13:56.641631    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:14:00.649213 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:00.649178    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:14:32.452666 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:32.452625    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:14:38.591420 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:38.591387    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"]
Sep 14 11:14:38.591894 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:38.591582    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8" podUID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerName="istio-proxy" containerID="cri-o://cc4667ac8212e916fc5a017af7716332bbdbc75195088fe4a8b61c71216afbb2" gracePeriod=30
Sep 14 11:14:43.851493 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.851462    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:14:43.924775 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.924744    2561 generic.go:356] "Generic (PLEG): container finished" podID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerID="cc4667ac8212e916fc5a017af7716332bbdbc75195088fe4a8b61c71216afbb2" exitCode=0
Sep 14 11:14:43.924934 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.924784    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"
Sep 14 11:14:43.924934 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.924825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8" event={"ID":"b24897dc-1da7-489f-ac43-b7515a99f29f","Type":"ContainerDied","Data":"cc4667ac8212e916fc5a017af7716332bbdbc75195088fe4a8b61c71216afbb2"}
Sep 14 11:14:43.924934 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.924875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8" event={"ID":"b24897dc-1da7-489f-ac43-b7515a99f29f","Type":"ContainerDied","Data":"8fcfd47ed43bd56440b5c90f4a8a8ad11b5b8d6df6ac69b2ebf77e9cf5192d43"}
Sep 14 11:14:43.924934 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.924899    2561 scope.go:117] "RemoveContainer" containerID="cc4667ac8212e916fc5a017af7716332bbdbc75195088fe4a8b61c71216afbb2"
Sep 14 11:14:43.934349 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.934331    2561 scope.go:117] "RemoveContainer" containerID="b6eac89989875cb84b34f6f6de64fa54d95096a4840f1ffb94f2c3c8f1acfdae"
Sep 14 11:14:43.939900 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.939878    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-envoy\") pod \"b24897dc-1da7-489f-ac43-b7515a99f29f\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") "
Sep 14 11:14:43.939991 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.939911    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-workload-socket\") pod \"b24897dc-1da7-489f-ac43-b7515a99f29f\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") "
Sep 14 11:14:43.939991 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.939946    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-podinfo\") pod \"b24897dc-1da7-489f-ac43-b7515a99f29f\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") "
Sep 14 11:14:43.940073 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.939997    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-data\") pod \"b24897dc-1da7-489f-ac43-b7515a99f29f\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") "
Sep 14 11:14:43.940073 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.940062    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24897dc-1da7-489f-ac43-b7515a99f29f-istiod-ca-cert\") pod \"b24897dc-1da7-489f-ac43-b7515a99f29f\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") "
Sep 14 11:14:43.940164 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.940115    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-token\") pod \"b24897dc-1da7-489f-ac43-b7515a99f29f\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") "
Sep 14 11:14:43.940164 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.940152    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-credential-socket\") pod \"b24897dc-1da7-489f-ac43-b7515a99f29f\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") "
Sep 14 11:14:43.940266 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.940179    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-workload-certs\") pod \"b24897dc-1da7-489f-ac43-b7515a99f29f\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") "
Sep 14 11:14:43.940266 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.940209    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b24897dc-1da7-489f-ac43-b7515a99f29f" (UID: "b24897dc-1da7-489f-ac43-b7515a99f29f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:14:43.940400 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.940369    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cp65d\" (UniqueName: \"kubernetes.io/projected/b24897dc-1da7-489f-ac43-b7515a99f29f-kube-api-access-cp65d\") pod \"b24897dc-1da7-489f-ac43-b7515a99f29f\" (UID: \"b24897dc-1da7-489f-ac43-b7515a99f29f\") "
Sep 14 11:14:43.940463 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.940391    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-data" (OuterVolumeSpecName: "istio-data") pod "b24897dc-1da7-489f-ac43-b7515a99f29f" (UID: "b24897dc-1da7-489f-ac43-b7515a99f29f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:14:43.940463 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.940414    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b24897dc-1da7-489f-ac43-b7515a99f29f" (UID: "b24897dc-1da7-489f-ac43-b7515a99f29f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:14:43.940765 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.940736    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b24897dc-1da7-489f-ac43-b7515a99f29f" (UID: "b24897dc-1da7-489f-ac43-b7515a99f29f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:14:43.940843 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.940806    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-workload-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:14:43.940843 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.940830    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-data\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:14:43.940955 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.940845    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-workload-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:14:43.941072 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.941047    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24897dc-1da7-489f-ac43-b7515a99f29f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b24897dc-1da7-489f-ac43-b7515a99f29f" (UID: "b24897dc-1da7-489f-ac43-b7515a99f29f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:14:43.942123 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.942098    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b24897dc-1da7-489f-ac43-b7515a99f29f" (UID: "b24897dc-1da7-489f-ac43-b7515a99f29f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 11:14:43.942413 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.942384    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b24897dc-1da7-489f-ac43-b7515a99f29f" (UID: "b24897dc-1da7-489f-ac43-b7515a99f29f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:14:43.942608 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.942586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24897dc-1da7-489f-ac43-b7515a99f29f-kube-api-access-cp65d" (OuterVolumeSpecName: "kube-api-access-cp65d") pod "b24897dc-1da7-489f-ac43-b7515a99f29f" (UID: "b24897dc-1da7-489f-ac43-b7515a99f29f"). InnerVolumeSpecName "kube-api-access-cp65d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:14:43.942741 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.942694    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-token" (OuterVolumeSpecName: "istio-token") pod "b24897dc-1da7-489f-ac43-b7515a99f29f" (UID: "b24897dc-1da7-489f-ac43-b7515a99f29f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:14:43.943491 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.943473    2561 scope.go:117] "RemoveContainer" containerID="cc4667ac8212e916fc5a017af7716332bbdbc75195088fe4a8b61c71216afbb2"
Sep 14 11:14:43.943789 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:14:43.943767    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4667ac8212e916fc5a017af7716332bbdbc75195088fe4a8b61c71216afbb2\": container with ID starting with cc4667ac8212e916fc5a017af7716332bbdbc75195088fe4a8b61c71216afbb2 not found: ID does not exist" containerID="cc4667ac8212e916fc5a017af7716332bbdbc75195088fe4a8b61c71216afbb2"
Sep 14 11:14:43.943841 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.943799    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4667ac8212e916fc5a017af7716332bbdbc75195088fe4a8b61c71216afbb2"} err="failed to get container status \"cc4667ac8212e916fc5a017af7716332bbdbc75195088fe4a8b61c71216afbb2\": rpc error: code = NotFound desc = could not find container \"cc4667ac8212e916fc5a017af7716332bbdbc75195088fe4a8b61c71216afbb2\": container with ID starting with cc4667ac8212e916fc5a017af7716332bbdbc75195088fe4a8b61c71216afbb2 not found: ID does not exist"
Sep 14 11:14:43.943841 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.943815    2561 scope.go:117] "RemoveContainer" containerID="b6eac89989875cb84b34f6f6de64fa54d95096a4840f1ffb94f2c3c8f1acfdae"
Sep 14 11:14:43.944104 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:14:43.944082    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6eac89989875cb84b34f6f6de64fa54d95096a4840f1ffb94f2c3c8f1acfdae\": container with ID starting with b6eac89989875cb84b34f6f6de64fa54d95096a4840f1ffb94f2c3c8f1acfdae not found: ID does not exist" containerID="b6eac89989875cb84b34f6f6de64fa54d95096a4840f1ffb94f2c3c8f1acfdae"
Sep 14 11:14:43.944144 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:43.944116    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6eac89989875cb84b34f6f6de64fa54d95096a4840f1ffb94f2c3c8f1acfdae"} err="failed to get container status \"b6eac89989875cb84b34f6f6de64fa54d95096a4840f1ffb94f2c3c8f1acfdae\": rpc error: code = NotFound desc = could not find container \"b6eac89989875cb84b34f6f6de64fa54d95096a4840f1ffb94f2c3c8f1acfdae\": container with ID starting with b6eac89989875cb84b34f6f6de64fa54d95096a4840f1ffb94f2c3c8f1acfdae not found: ID does not exist"
Sep 14 11:14:44.041748 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:44.041707    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24897dc-1da7-489f-ac43-b7515a99f29f-istiod-ca-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:14:44.041748 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:44.041741    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:14:44.041748 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:44.041749    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-credential-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:14:44.041928 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:44.041760    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp65d\" (UniqueName: \"kubernetes.io/projected/b24897dc-1da7-489f-ac43-b7515a99f29f-kube-api-access-cp65d\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:14:44.041928 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:44.041807    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-envoy\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:14:44.041928 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:44.041820    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24897dc-1da7-489f-ac43-b7515a99f29f-istio-podinfo\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:14:44.268830 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:44.268801    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"]
Sep 14 11:14:44.280845 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:44.280816    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e6e0b4a3-openshift-default-8cc4d8bd8-ns5x8"]
Sep 14 11:14:45.803829 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:14:45.803797    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b24897dc-1da7-489f-ac43-b7515a99f29f" path="/var/lib/kubelet/pods/b24897dc-1da7-489f-ac43-b7515a99f29f/volumes"
Sep 14 11:15:00.258163 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:00.258127    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm"]
Sep 14 11:15:00.258668 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:00.258575    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerName="istio-proxy"
Sep 14 11:15:00.258668 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:00.258586    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerName="istio-proxy"
Sep 14 11:15:00.258668 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:00.258602    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerName="istio-proxy"
Sep 14 11:15:00.258668 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:00.258608    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerName="istio-proxy"
Sep 14 11:15:00.258668 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:00.258667    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerName="istio-proxy"
Sep 14 11:15:00.261875 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:00.261854    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm"
Sep 14 11:15:00.272297 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:00.272270    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\""
Sep 14 11:15:00.272462 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:00.272444    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm"
Sep 14 11:15:00.282723 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:00.282685    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm"]
Sep 14 11:15:00.568577 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:00.568544    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm"]
Sep 14 11:15:00.569230 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:15:00.569196    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb12f249_790a_45d1_a2bb_c8a8103c8131.slice/crio-3f9a688ecdb6556fb36114612ed26af2ac8804cfd7bffa708899d67329f17291 WatchSource:0}: Error finding container 3f9a688ecdb6556fb36114612ed26af2ac8804cfd7bffa708899d67329f17291: Status 404 returned error can't find the container with id 3f9a688ecdb6556fb36114612ed26af2ac8804cfd7bffa708899d67329f17291
Sep 14 11:15:00.571529 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:00.571509    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 11:15:01.037389 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:01.037356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm" event={"ID":"eb12f249-790a-45d1-a2bb-c8a8103c8131","Type":"ContainerStarted","Data":"3f9a688ecdb6556fb36114612ed26af2ac8804cfd7bffa708899d67329f17291"}
Sep 14 11:15:04.059775 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:04.059745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm" event={"ID":"eb12f249-790a-45d1-a2bb-c8a8103c8131","Type":"ContainerStarted","Data":"7a59b6c9a48aaca3ae20ee0212c6161b3105163999db153937330bf76bb7b925"}
Sep 14 11:15:04.085413 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:04.085361    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm" podStartSLOduration=1.545345676 podStartE2EDuration="4.085344895s" totalImagesPullingTime="2.539999219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:04.083162119 +0000 UTC m=+1608.767846535" watchObservedRunningTime="2026-09-14 11:15:04.085344895 +0000 UTC m=+1608.770029305"
Sep 14 11:15:05.068553 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:05.068519    2561 generic.go:356] "Generic (PLEG): container finished" podID="eb12f249-790a-45d1-a2bb-c8a8103c8131" containerID="7a59b6c9a48aaca3ae20ee0212c6161b3105163999db153937330bf76bb7b925" exitCode=0
Sep 14 11:15:05.068967 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:05.068608    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm" event={"ID":"eb12f249-790a-45d1-a2bb-c8a8103c8131","Type":"ContainerDied","Data":"7a59b6c9a48aaca3ae20ee0212c6161b3105163999db153937330bf76bb7b925"}
Sep 14 11:15:06.212425 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:06.212401    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm"
Sep 14 11:15:07.082008 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:07.081979    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm"
Sep 14 11:15:07.082185 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:07.082005    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm" event={"ID":"eb12f249-790a-45d1-a2bb-c8a8103c8131","Type":"ContainerDied","Data":"3f9a688ecdb6556fb36114612ed26af2ac8804cfd7bffa708899d67329f17291"}
Sep 14 11:15:07.082185 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:07.082051    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f9a688ecdb6556fb36114612ed26af2ac8804cfd7bffa708899d67329f17291"
Sep 14 11:15:10.338264 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:10.338188    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:15:24.070600 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:24.070566    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"]
Sep 14 11:15:38.340277 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:38.340243    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"]
Sep 14 11:15:39.090125 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.090090    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v" podUID="1cb73c64-12a0-4d06-bc68-d94f69c6c7ab" containerName="main" containerID="cri-o://65269aae114e88dca920cdd17907d834b9dc8d5d5f23dcd6565a71ffc5db4f6e" gracePeriod=45
Sep 14 11:15:39.290011 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.289973    2561 generic.go:356] "Generic (PLEG): container finished" podID="1cb73c64-12a0-4d06-bc68-d94f69c6c7ab" containerID="65269aae114e88dca920cdd17907d834b9dc8d5d5f23dcd6565a71ffc5db4f6e" exitCode=0
Sep 14 11:15:39.290168 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.290081    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v" event={"ID":"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab","Type":"ContainerDied","Data":"65269aae114e88dca920cdd17907d834b9dc8d5d5f23dcd6565a71ffc5db4f6e"}
Sep 14 11:15:39.378619 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.378590    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:15:39.486385 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.486354    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-tls-certs\") pod \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") "
Sep 14 11:15:39.486547 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.486429    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-tmp-dir\") pod \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") "
Sep 14 11:15:39.486547 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.486468    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-home\") pod \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") "
Sep 14 11:15:39.486547 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.486501    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-model-cache\") pod \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") "
Sep 14 11:15:39.486676 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.486561    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zczvn\" (UniqueName: \"kubernetes.io/projected/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-kube-api-access-zczvn\") pod \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") "
Sep 14 11:15:39.486756 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.486676    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-dshm\") pod \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\" (UID: \"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab\") "
Sep 14 11:15:39.486981 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.486940    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-home" (OuterVolumeSpecName: "home") pod "1cb73c64-12a0-4d06-bc68-d94f69c6c7ab" (UID: "1cb73c64-12a0-4d06-bc68-d94f69c6c7ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:15:39.487155 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.487127    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "1cb73c64-12a0-4d06-bc68-d94f69c6c7ab" (UID: "1cb73c64-12a0-4d06-bc68-d94f69c6c7ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:15:39.487329 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.487296    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1cb73c64-12a0-4d06-bc68-d94f69c6c7ab" (UID: "1cb73c64-12a0-4d06-bc68-d94f69c6c7ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:15:39.489275 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.489245    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cb73c64-12a0-4d06-bc68-d94f69c6c7ab" (UID: "1cb73c64-12a0-4d06-bc68-d94f69c6c7ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:15:39.489275 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.489265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-kube-api-access-zczvn" (OuterVolumeSpecName: "kube-api-access-zczvn") pod "1cb73c64-12a0-4d06-bc68-d94f69c6c7ab" (UID: "1cb73c64-12a0-4d06-bc68-d94f69c6c7ab"). InnerVolumeSpecName "kube-api-access-zczvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:15:39.489587 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.489563    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-dshm" (OuterVolumeSpecName: "dshm") pod "1cb73c64-12a0-4d06-bc68-d94f69c6c7ab" (UID: "1cb73c64-12a0-4d06-bc68-d94f69c6c7ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:15:39.588123 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.588087    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-tls-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:39.588123 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.588121    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-tmp-dir\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:39.588123 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.588131    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-home\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:39.588344 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.588142    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-model-cache\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:39.588344 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.588153    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zczvn\" (UniqueName: \"kubernetes.io/projected/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-kube-api-access-zczvn\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:39.588344 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:39.588165    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab-dshm\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:40.305492 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:40.305447    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v" event={"ID":"1cb73c64-12a0-4d06-bc68-d94f69c6c7ab","Type":"ContainerDied","Data":"f4992a1ca74854e768f4e5677a57fd2624dccff7995933400124e0d3598277c5"}
Sep 14 11:15:40.305668 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:40.305527    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"
Sep 14 11:15:40.305668 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:40.305508    2561 scope.go:117] "RemoveContainer" containerID="65269aae114e88dca920cdd17907d834b9dc8d5d5f23dcd6565a71ffc5db4f6e"
Sep 14 11:15:40.369668 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:40.369638    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"]
Sep 14 11:15:40.379381 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:40.379359    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/embedding-w0-7984b1-kserve-84c6bf55dc-7ms4v"]
Sep 14 11:15:41.803360 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:41.803329    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cb73c64-12a0-4d06-bc68-d94f69c6c7ab" path="/var/lib/kubelet/pods/1cb73c64-12a0-4d06-bc68-d94f69c6c7ab/volumes"
Sep 14 11:15:42.366232 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.366193    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"]
Sep 14 11:15:42.366843 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.366822    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cb73c64-12a0-4d06-bc68-d94f69c6c7ab" containerName="main"
Sep 14 11:15:42.366945 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.366847    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cb73c64-12a0-4d06-bc68-d94f69c6c7ab" containerName="main"
Sep 14 11:15:42.366945 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.366880    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb12f249-790a-45d1-a2bb-c8a8103c8131" containerName="cleanup"
Sep 14 11:15:42.366945 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.366891    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb12f249-790a-45d1-a2bb-c8a8103c8131" containerName="cleanup"
Sep 14 11:15:42.367051 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.367016    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb12f249-790a-45d1-a2bb-c8a8103c8131" containerName="cleanup"
Sep 14 11:15:42.367051 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.367035    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b24897dc-1da7-489f-ac43-b7515a99f29f" containerName="istio-proxy"
Sep 14 11:15:42.367051 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.367047    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cb73c64-12a0-4d06-bc68-d94f69c6c7ab" containerName="main"
Sep 14 11:15:42.371831 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.371807    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.375134 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.375114    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-29e74e-serving-cert\""
Sep 14 11:15:42.404235 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.404209    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"]
Sep 14 11:15:42.516647 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.516617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-metrics-certs\") pod \"maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.516863 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.516652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hhmd\" (UniqueName: \"kubernetes.io/projected/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-kube-api-access-7hhmd\") pod \"maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.516863 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.516808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-maas-api-tls\") pod \"maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.516975 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.516868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-openshift-service-ca\") pod \"maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.617781 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.617672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-maas-api-tls\") pod \"maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.617933 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.617825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-openshift-service-ca\") pod \"maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.617933 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.617874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-metrics-certs\") pod \"maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.617933 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.617898    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hhmd\" (UniqueName: \"kubernetes.io/projected/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-kube-api-access-7hhmd\") pod \"maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.618533 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.618500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-openshift-service-ca\") pod \"maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.620220 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.620197    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-maas-api-tls\") pod \"maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.620342 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.620321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-metrics-certs\") pod \"maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.628703 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.628680    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hhmd\" (UniqueName: \"kubernetes.io/projected/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-kube-api-access-7hhmd\") pod \"maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.682620 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.682589    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:42.961733 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:42.961687    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"]
Sep 14 11:15:42.963271 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:15:42.963227    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod682102ab_c669_4efd_a4a6_b3fdd1c5ab5f.slice/crio-504b76eca52dd32f4d43277f8813425a7243156f6d2792b18c2338d21564bdd8 WatchSource:0}: Error finding container 504b76eca52dd32f4d43277f8813425a7243156f6d2792b18c2338d21564bdd8: Status 404 returned error can't find the container with id 504b76eca52dd32f4d43277f8813425a7243156f6d2792b18c2338d21564bdd8
Sep 14 11:15:43.325462 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:43.325375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25" event={"ID":"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f","Type":"ContainerStarted","Data":"504b76eca52dd32f4d43277f8813425a7243156f6d2792b18c2338d21564bdd8"}
Sep 14 11:15:44.333066 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:44.333033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25" event={"ID":"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f","Type":"ContainerStarted","Data":"7c3bba8f1a3bb3d67d7122d66270344cdb8ff58e5242f65683299cafecbc9ef2"}
Sep 14 11:15:44.333424 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:44.333093    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:44.363836 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:44.363787    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25" podStartSLOduration=1.638332343 podStartE2EDuration="2.363772537s" totalImagesPullingTime="725.440194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:44.361125309 +0000 UTC m=+1649.045809710" watchObservedRunningTime="2026-09-14 11:15:44.363772537 +0000 UTC m=+1649.048456926"
Sep 14 11:15:48.365210 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:48.365176    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"]
Sep 14 11:15:48.365665 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:48.365450    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerName="istio-proxy" containerID="cri-o://5f84a30ba15cbde4d71251dc0cfb75d10bea2f28aa7693556a2b83775bb045ac" gracePeriod=30
Sep 14 11:15:50.052760 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:50.052705    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:15:50.345407 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:50.345314    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:15:52.437836 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:52.437800    2561 patch_prober.go:28] interesting pod/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 11:15:52.438282 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:52.437892    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 11:15:53.359061 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.359017    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962" podUID="a474c605-06a5-418c-a248-6236c629c6e9" containerName="main" containerID="cri-o://2ce13cbfd7ab8900d8bcd8cb23601887e8e4c5c5387070054a837d2641f5681a" gracePeriod=45
Sep 14 11:15:53.633574 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.633553    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:15:53.637296 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.637276    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:15:53.727304 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727271    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-podinfo\") pod \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") "
Sep 14 11:15:53.727304 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727310    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-dshm\") pod \"a474c605-06a5-418c-a248-6236c629c6e9\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") "
Sep 14 11:15:53.727525 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727335    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-data\") pod \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") "
Sep 14 11:15:53.727525 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727359    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-envoy\") pod \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") "
Sep 14 11:15:53.727525 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727400    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istiod-ca-cert\") pod \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") "
Sep 14 11:15:53.727525 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727435    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-home\") pod \"a474c605-06a5-418c-a248-6236c629c6e9\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") "
Sep 14 11:15:53.727525 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727479    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-workload-certs\") pod \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") "
Sep 14 11:15:53.727834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727559    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-token\") pod \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") "
Sep 14 11:15:53.727834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727603    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-workload-socket\") pod \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") "
Sep 14 11:15:53.727834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727646    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-tmp-dir\") pod \"a474c605-06a5-418c-a248-6236c629c6e9\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") "
Sep 14 11:15:53.727834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727662    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-data" (OuterVolumeSpecName: "istio-data") pod "f51381b7-acf4-4af2-b15a-1dc7119d2c6c" (UID: "f51381b7-acf4-4af2-b15a-1dc7119d2c6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:15:53.727834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727691    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz66v\" (UniqueName: \"kubernetes.io/projected/a474c605-06a5-418c-a248-6236c629c6e9-kube-api-access-fz66v\") pod \"a474c605-06a5-418c-a248-6236c629c6e9\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") "
Sep 14 11:15:53.727834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727766    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-credential-socket\") pod \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") "
Sep 14 11:15:53.727834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727793    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a474c605-06a5-418c-a248-6236c629c6e9-tls-certs\") pod \"a474c605-06a5-418c-a248-6236c629c6e9\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") "
Sep 14 11:15:53.728309 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727875    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22sbl\" (UniqueName: \"kubernetes.io/projected/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-kube-api-access-22sbl\") pod \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\" (UID: \"f51381b7-acf4-4af2-b15a-1dc7119d2c6c\") "
Sep 14 11:15:53.728309 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727907    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-model-cache\") pod \"a474c605-06a5-418c-a248-6236c629c6e9\" (UID: \"a474c605-06a5-418c-a248-6236c629c6e9\") "
Sep 14 11:15:53.728309 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727934    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f51381b7-acf4-4af2-b15a-1dc7119d2c6c" (UID: "f51381b7-acf4-4af2-b15a-1dc7119d2c6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:15:53.728309 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.727948    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f51381b7-acf4-4af2-b15a-1dc7119d2c6c" (UID: "f51381b7-acf4-4af2-b15a-1dc7119d2c6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:15:53.728309 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.728209    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f51381b7-acf4-4af2-b15a-1dc7119d2c6c" (UID: "f51381b7-acf4-4af2-b15a-1dc7119d2c6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:15:53.728593 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.728320    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f51381b7-acf4-4af2-b15a-1dc7119d2c6c" (UID: "f51381b7-acf4-4af2-b15a-1dc7119d2c6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:15:53.728593 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.728440    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-workload-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.728593 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.728464    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-workload-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.728593 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.728481    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-credential-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.728593 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.728498    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-data\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.728869 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.728631    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-home" (OuterVolumeSpecName: "home") pod "a474c605-06a5-418c-a248-6236c629c6e9" (UID: "a474c605-06a5-418c-a248-6236c629c6e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:15:53.728869 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.728676    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a474c605-06a5-418c-a248-6236c629c6e9" (UID: "a474c605-06a5-418c-a248-6236c629c6e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:15:53.729114 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.729061    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a474c605-06a5-418c-a248-6236c629c6e9" (UID: "a474c605-06a5-418c-a248-6236c629c6e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:15:53.730359 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.730327    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f51381b7-acf4-4af2-b15a-1dc7119d2c6c" (UID: "f51381b7-acf4-4af2-b15a-1dc7119d2c6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:15:53.730479 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.730422    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f51381b7-acf4-4af2-b15a-1dc7119d2c6c" (UID: "f51381b7-acf4-4af2-b15a-1dc7119d2c6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 11:15:53.730837 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.730811    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-dshm" (OuterVolumeSpecName: "dshm") pod "a474c605-06a5-418c-a248-6236c629c6e9" (UID: "a474c605-06a5-418c-a248-6236c629c6e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:15:53.730986 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.730946    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a474c605-06a5-418c-a248-6236c629c6e9-kube-api-access-fz66v" (OuterVolumeSpecName: "kube-api-access-fz66v") pod "a474c605-06a5-418c-a248-6236c629c6e9" (UID: "a474c605-06a5-418c-a248-6236c629c6e9"). InnerVolumeSpecName "kube-api-access-fz66v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:15:53.731277 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.731257    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-token" (OuterVolumeSpecName: "istio-token") pod "f51381b7-acf4-4af2-b15a-1dc7119d2c6c" (UID: "f51381b7-acf4-4af2-b15a-1dc7119d2c6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:15:53.731383 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.731281    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-kube-api-access-22sbl" (OuterVolumeSpecName: "kube-api-access-22sbl") pod "f51381b7-acf4-4af2-b15a-1dc7119d2c6c" (UID: "f51381b7-acf4-4af2-b15a-1dc7119d2c6c"). InnerVolumeSpecName "kube-api-access-22sbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:15:53.732000 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.731975    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a474c605-06a5-418c-a248-6236c629c6e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a474c605-06a5-418c-a248-6236c629c6e9" (UID: "a474c605-06a5-418c-a248-6236c629c6e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:15:53.829092 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.829070    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22sbl\" (UniqueName: \"kubernetes.io/projected/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-kube-api-access-22sbl\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.829092 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.829094    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-model-cache\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.829225 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.829103    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-podinfo\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.829225 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.829112    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-dshm\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.829225 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.829119    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-envoy\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.829225 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.829127    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istiod-ca-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.829225 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.829134    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-home\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.829225 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.829141    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51381b7-acf4-4af2-b15a-1dc7119d2c6c-istio-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.829225 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.829148    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a474c605-06a5-418c-a248-6236c629c6e9-tmp-dir\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.829225 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.829156    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz66v\" (UniqueName: \"kubernetes.io/projected/a474c605-06a5-418c-a248-6236c629c6e9-kube-api-access-fz66v\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:53.829225 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:53.829165    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a474c605-06a5-418c-a248-6236c629c6e9-tls-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:15:54.397641 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.397610    2561 generic.go:356] "Generic (PLEG): container finished" podID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerID="5f84a30ba15cbde4d71251dc0cfb75d10bea2f28aa7693556a2b83775bb045ac" exitCode=0
Sep 14 11:15:54.397796 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.397666    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"
Sep 14 11:15:54.397796 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.397686    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" event={"ID":"f51381b7-acf4-4af2-b15a-1dc7119d2c6c","Type":"ContainerDied","Data":"5f84a30ba15cbde4d71251dc0cfb75d10bea2f28aa7693556a2b83775bb045ac"}
Sep 14 11:15:54.397796 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.397748    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc" event={"ID":"f51381b7-acf4-4af2-b15a-1dc7119d2c6c","Type":"ContainerDied","Data":"4d1a653f1ff787cdbf0f479caf8f150a0f8727b4b51db6a947fe21c04bd2ea8c"}
Sep 14 11:15:54.397796 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.397770    2561 scope.go:117] "RemoveContainer" containerID="5f84a30ba15cbde4d71251dc0cfb75d10bea2f28aa7693556a2b83775bb045ac"
Sep 14 11:15:54.399242 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.399224    2561 generic.go:356] "Generic (PLEG): container finished" podID="a474c605-06a5-418c-a248-6236c629c6e9" containerID="2ce13cbfd7ab8900d8bcd8cb23601887e8e4c5c5387070054a837d2641f5681a" exitCode=0
Sep 14 11:15:54.399335 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.399279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962" event={"ID":"a474c605-06a5-418c-a248-6236c629c6e9","Type":"ContainerDied","Data":"2ce13cbfd7ab8900d8bcd8cb23601887e8e4c5c5387070054a837d2641f5681a"}
Sep 14 11:15:54.399335 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.399290    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"
Sep 14 11:15:54.399335 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.399309    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962" event={"ID":"a474c605-06a5-418c-a248-6236c629c6e9","Type":"ContainerDied","Data":"7b25d58063a73618fc1c23302860d13f69300398aa86cff51897f7b8d222c64e"}
Sep 14 11:15:54.407291 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.407276    2561 scope.go:117] "RemoveContainer" containerID="44c6aecdf266989e29021f14b68dfe1e0a8aac4c9f2d52e46af12ac452adb2d1"
Sep 14 11:15:54.419680 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.419662    2561 scope.go:117] "RemoveContainer" containerID="5f84a30ba15cbde4d71251dc0cfb75d10bea2f28aa7693556a2b83775bb045ac"
Sep 14 11:15:54.420003 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:15:54.419983    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f84a30ba15cbde4d71251dc0cfb75d10bea2f28aa7693556a2b83775bb045ac\": container with ID starting with 5f84a30ba15cbde4d71251dc0cfb75d10bea2f28aa7693556a2b83775bb045ac not found: ID does not exist" containerID="5f84a30ba15cbde4d71251dc0cfb75d10bea2f28aa7693556a2b83775bb045ac"
Sep 14 11:15:54.420069 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.420011    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f84a30ba15cbde4d71251dc0cfb75d10bea2f28aa7693556a2b83775bb045ac"} err="failed to get container status \"5f84a30ba15cbde4d71251dc0cfb75d10bea2f28aa7693556a2b83775bb045ac\": rpc error: code = NotFound desc = could not find container \"5f84a30ba15cbde4d71251dc0cfb75d10bea2f28aa7693556a2b83775bb045ac\": container with ID starting with 5f84a30ba15cbde4d71251dc0cfb75d10bea2f28aa7693556a2b83775bb045ac not found: ID does not exist"
Sep 14 11:15:54.420069 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.420029    2561 scope.go:117] "RemoveContainer" containerID="44c6aecdf266989e29021f14b68dfe1e0a8aac4c9f2d52e46af12ac452adb2d1"
Sep 14 11:15:54.420344 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:15:54.420320    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c6aecdf266989e29021f14b68dfe1e0a8aac4c9f2d52e46af12ac452adb2d1\": container with ID starting with 44c6aecdf266989e29021f14b68dfe1e0a8aac4c9f2d52e46af12ac452adb2d1 not found: ID does not exist" containerID="44c6aecdf266989e29021f14b68dfe1e0a8aac4c9f2d52e46af12ac452adb2d1"
Sep 14 11:15:54.420451 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.420351    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c6aecdf266989e29021f14b68dfe1e0a8aac4c9f2d52e46af12ac452adb2d1"} err="failed to get container status \"44c6aecdf266989e29021f14b68dfe1e0a8aac4c9f2d52e46af12ac452adb2d1\": rpc error: code = NotFound desc = could not find container \"44c6aecdf266989e29021f14b68dfe1e0a8aac4c9f2d52e46af12ac452adb2d1\": container with ID starting with 44c6aecdf266989e29021f14b68dfe1e0a8aac4c9f2d52e46af12ac452adb2d1 not found: ID does not exist"
Sep 14 11:15:54.420451 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.420372    2561 scope.go:117] "RemoveContainer" containerID="2ce13cbfd7ab8900d8bcd8cb23601887e8e4c5c5387070054a837d2641f5681a"
Sep 14 11:15:54.422513 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.422490    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"]
Sep 14 11:15:54.425206 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.425187    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7984b1-openshift-default-74dd5fd5d6-26dnc"]
Sep 14 11:15:54.428992 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.428976    2561 scope.go:117] "RemoveContainer" containerID="2ce13cbfd7ab8900d8bcd8cb23601887e8e4c5c5387070054a837d2641f5681a"
Sep 14 11:15:54.429273 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:15:54.429256    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce13cbfd7ab8900d8bcd8cb23601887e8e4c5c5387070054a837d2641f5681a\": container with ID starting with 2ce13cbfd7ab8900d8bcd8cb23601887e8e4c5c5387070054a837d2641f5681a not found: ID does not exist" containerID="2ce13cbfd7ab8900d8bcd8cb23601887e8e4c5c5387070054a837d2641f5681a"
Sep 14 11:15:54.429318 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.429281    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce13cbfd7ab8900d8bcd8cb23601887e8e4c5c5387070054a837d2641f5681a"} err="failed to get container status \"2ce13cbfd7ab8900d8bcd8cb23601887e8e4c5c5387070054a837d2641f5681a\": rpc error: code = NotFound desc = could not find container \"2ce13cbfd7ab8900d8bcd8cb23601887e8e4c5c5387070054a837d2641f5681a\": container with ID starting with 2ce13cbfd7ab8900d8bcd8cb23601887e8e4c5c5387070054a837d2641f5681a not found: ID does not exist"
Sep 14 11:15:54.439739 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.439706    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"]
Sep 14 11:15:54.443909 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:54.443890    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/simulator-w4-4ebf73-kserve-6d98946478-ml962"]
Sep 14 11:15:55.803323 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:55.803281    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a474c605-06a5-418c-a248-6236c629c6e9" path="/var/lib/kubelet/pods/a474c605-06a5-418c-a248-6236c629c6e9/volumes"
Sep 14 11:15:55.803933 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:55.803911    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" path="/var/lib/kubelet/pods/f51381b7-acf4-4af2-b15a-1dc7119d2c6c/volumes"
Sep 14 11:15:56.241500 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.241466    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"]
Sep 14 11:15:56.242147 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.242132    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerName="istio-proxy"
Sep 14 11:15:56.242199 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.242151    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerName="istio-proxy"
Sep 14 11:15:56.242199 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.242179    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerName="istio-proxy"
Sep 14 11:15:56.242199 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.242188    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerName="istio-proxy"
Sep 14 11:15:56.242300 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.242205    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a474c605-06a5-418c-a248-6236c629c6e9" containerName="main"
Sep 14 11:15:56.242300 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.242215    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a474c605-06a5-418c-a248-6236c629c6e9" containerName="main"
Sep 14 11:15:56.242363 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.242309    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerName="istio-proxy"
Sep 14 11:15:56.242363 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.242321    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a474c605-06a5-418c-a248-6236c629c6e9" containerName="main"
Sep 14 11:15:56.247691 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.247670    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.263555 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.263536    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f8e582-openshift-default-dockercfg-p8x8c\""
Sep 14 11:15:56.287086 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.287059    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"]
Sep 14 11:15:56.353287 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.353246    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istiod-ca-cert\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.353499 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.353355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-credential-socket\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.353499 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.353469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-token\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.353640 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.353538    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-podinfo\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.353640 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.353577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz4dt\" (UniqueName: \"kubernetes.io/projected/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-kube-api-access-qz4dt\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.353640 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.353634    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-envoy\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.353847 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.353659    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-workload-certs\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.353847 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.353705    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-workload-socket\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.353847 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.353744    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-data\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.455477 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.455442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-credential-socket\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.455648 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.455507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-token\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.455648 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.455557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-podinfo\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.455648 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.455582    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz4dt\" (UniqueName: \"kubernetes.io/projected/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-kube-api-access-qz4dt\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.455648 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.455604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-envoy\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.455925 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.455737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-workload-certs\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.455925 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.455775    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-workload-socket\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.455925 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.455797    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-data\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.456092 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.456031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-workload-certs\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.456229 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.456204    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-data\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.456304 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.456264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istiod-ca-cert\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.456372 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.456267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-workload-socket\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.456477 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.456459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-credential-socket\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.456810 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.456793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istiod-ca-cert\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.458564 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.458531    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-envoy\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.458653 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.458637    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-podinfo\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.484003 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.483975    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz4dt\" (UniqueName: \"kubernetes.io/projected/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-kube-api-access-qz4dt\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.485120 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.485100    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-token\") pod \"e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") " pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.558938 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.558882    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:56.846175 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:15:56.846135    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947e8ad6_e7dc_4740_a0e7_7a05962f91f3.slice/crio-2e428bcd6d3968f83af6559b591c23ad289d67131df363d372fb318f425dc4b9 WatchSource:0}: Error finding container 2e428bcd6d3968f83af6559b591c23ad289d67131df363d372fb318f425dc4b9: Status 404 returned error can't find the container with id 2e428bcd6d3968f83af6559b591c23ad289d67131df363d372fb318f425dc4b9
Sep 14 11:15:56.846591 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.846269    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"]
Sep 14 11:15:56.848615 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.848578    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:15:56.848733 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.848659    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:15:56.848733 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:56.848698    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:15:57.422505 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:57.422466    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt" event={"ID":"947e8ad6-e7dc-4740-a0e7-7a05962f91f3","Type":"ContainerStarted","Data":"b3c0dae808ff4e224653e69b400c12902365cc917c32444dcbc78dc01e88e6a6"}
Sep 14 11:15:57.422699 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:57.422528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt" event={"ID":"947e8ad6-e7dc-4740-a0e7-7a05962f91f3","Type":"ContainerStarted","Data":"2e428bcd6d3968f83af6559b591c23ad289d67131df363d372fb318f425dc4b9"}
Sep 14 11:15:57.456688 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:57.456634    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt" podStartSLOduration=1.456620612 podStartE2EDuration="1.456620612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:57.455077039 +0000 UTC m=+1662.139761440" watchObservedRunningTime="2026-09-14 11:15:57.456620612 +0000 UTC m=+1662.141305001"
Sep 14 11:15:57.559944 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:57.559911    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:57.565423 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:57.565395    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:58.428350 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:58.428318    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:15:58.429389 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:15:58.429367    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:16:06.906390 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:06.906354    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"]
Sep 14 11:16:06.906905 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:06.906890    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f51381b7-acf4-4af2-b15a-1dc7119d2c6c" containerName="istio-proxy"
Sep 14 11:16:06.910189 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:06.910172    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:06.913421 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:06.913393    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f8e582-serving-cert\""
Sep 14 11:16:06.936256 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:06.936229    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"]
Sep 14 11:16:06.958574 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:06.958544    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-metrics-certs\") pod \"maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:06.958704 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:06.958604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc231966-027b-4833-aba3-1db6f05da322-openshift-service-ca\") pod \"maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:06.958704 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:06.958651    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-maas-api-tls\") pod \"maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:06.958704 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:06.958675    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twl5c\" (UniqueName: \"kubernetes.io/projected/dc231966-027b-4833-aba3-1db6f05da322-kube-api-access-twl5c\") pod \"maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:07.060048 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:07.060009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-metrics-certs\") pod \"maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:07.060257 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:07.060075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc231966-027b-4833-aba3-1db6f05da322-openshift-service-ca\") pod \"maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:07.060257 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:07.060103    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-maas-api-tls\") pod \"maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:07.060257 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:07.060123    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twl5c\" (UniqueName: \"kubernetes.io/projected/dc231966-027b-4833-aba3-1db6f05da322-kube-api-access-twl5c\") pod \"maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:07.060257 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:16:07.060249    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-serving-cert: secret "maas-api-e2e-api-b-f8e582-serving-cert" not found
Sep 14 11:16:07.060468 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:16:07.060323    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-maas-api-tls podName:dc231966-027b-4833-aba3-1db6f05da322 nodeName:}" failed. No retries permitted until 2026-09-14 11:16:07.560304595 +0000 UTC m=+1672.244988975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-maas-api-tls") pod "maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49" (UID: "dc231966-027b-4833-aba3-1db6f05da322") : secret "maas-api-e2e-api-b-f8e582-serving-cert" not found
Sep 14 11:16:07.060763 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:07.060741    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc231966-027b-4833-aba3-1db6f05da322-openshift-service-ca\") pod \"maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:07.062451 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:07.062428    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-metrics-certs\") pod \"maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:07.083996 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:07.083962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twl5c\" (UniqueName: \"kubernetes.io/projected/dc231966-027b-4833-aba3-1db6f05da322-kube-api-access-twl5c\") pod \"maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:07.566420 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:07.566378    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-maas-api-tls\") pod \"maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:07.568816 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:07.568795    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-maas-api-tls\") pod \"maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:07.821508 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:07.821409    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:07.823542 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:07.823520    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"]
Sep 14 11:16:07.823792 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:07.823747    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy" containerID="cri-o://d5365beae84a8cacf12b797ae21a6e262ab392bcc83440205581305eaef7c4c1" gracePeriod=30
Sep 14 11:16:08.128838 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:16:08.128804    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc231966_027b_4833_aba3_1db6f05da322.slice/crio-e9597fb29a4f6a9aa0baf5a57b5c84f89af3114bd7cd93bc27d6c0fa668d720a WatchSource:0}: Error finding container e9597fb29a4f6a9aa0baf5a57b5c84f89af3114bd7cd93bc27d6c0fa668d720a: Status 404 returned error can't find the container with id e9597fb29a4f6a9aa0baf5a57b5c84f89af3114bd7cd93bc27d6c0fa668d720a
Sep 14 11:16:08.129253 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:08.128870    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"]
Sep 14 11:16:08.495668 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:08.495592    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49" event={"ID":"dc231966-027b-4833-aba3-1db6f05da322","Type":"ContainerStarted","Data":"e9597fb29a4f6a9aa0baf5a57b5c84f89af3114bd7cd93bc27d6c0fa668d720a"}
Sep 14 11:16:09.503679 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:09.503638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49" event={"ID":"dc231966-027b-4833-aba3-1db6f05da322","Type":"ContainerStarted","Data":"9fd930715b592d1763508640db59a09c20b77467aa7b7975d9e25cfd77355715"}
Sep 14 11:16:09.504123 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:09.503852    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:09.523602 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:09.523570    2561 patch_prober.go:28] interesting pod/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 11:16:09.523768 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:09.523616    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 11:16:09.544996 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:09.544949    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49" podStartSLOduration=3.157385635 podStartE2EDuration="3.544934135s" totalImagesPullingTime="387.5485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:16:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:16:09.542673721 +0000 UTC m=+1674.227358109" watchObservedRunningTime="2026-09-14 11:16:09.544934135 +0000 UTC m=+1674.229618575"
Sep 14 11:16:13.087707 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.087647    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:16:13.123216 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.123191    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-podinfo\") pod \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") "
Sep 14 11:16:13.123355 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.123226    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-workload-socket\") pod \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") "
Sep 14 11:16:13.123355 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.123244    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-data\") pod \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") "
Sep 14 11:16:13.123355 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.123278    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-token\") pod \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") "
Sep 14 11:16:13.123355 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.123304    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-workload-certs\") pod \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") "
Sep 14 11:16:13.123355 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.123332    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-envoy\") pod \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") "
Sep 14 11:16:13.123639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.123402    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istiod-ca-cert\") pod \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") "
Sep 14 11:16:13.123639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.123449    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-credential-socket\") pod \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") "
Sep 14 11:16:13.123639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.123513    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg8nn\" (UniqueName: \"kubernetes.io/projected/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-kube-api-access-vg8nn\") pod \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\" (UID: \"0d6291a6-7a72-48f0-bf8c-80dd9ed61138\") "
Sep 14 11:16:13.123639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.123531    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0d6291a6-7a72-48f0-bf8c-80dd9ed61138" (UID: "0d6291a6-7a72-48f0-bf8c-80dd9ed61138"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:16:13.123639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.123560    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-data" (OuterVolumeSpecName: "istio-data") pod "0d6291a6-7a72-48f0-bf8c-80dd9ed61138" (UID: "0d6291a6-7a72-48f0-bf8c-80dd9ed61138"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:16:13.123930 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.123780    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0d6291a6-7a72-48f0-bf8c-80dd9ed61138" (UID: "0d6291a6-7a72-48f0-bf8c-80dd9ed61138"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:16:13.123930 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.123884    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0d6291a6-7a72-48f0-bf8c-80dd9ed61138" (UID: "0d6291a6-7a72-48f0-bf8c-80dd9ed61138"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:16:13.126658 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.124495    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-workload-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:16:13.126658 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.124542    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-data\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:16:13.126658 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.124559    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-workload-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:16:13.126658 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.125240    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0d6291a6-7a72-48f0-bf8c-80dd9ed61138" (UID: "0d6291a6-7a72-48f0-bf8c-80dd9ed61138"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:16:13.130333 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.127305    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-token" (OuterVolumeSpecName: "istio-token") pod "0d6291a6-7a72-48f0-bf8c-80dd9ed61138" (UID: "0d6291a6-7a72-48f0-bf8c-80dd9ed61138"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:16:13.130333 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.127800    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0d6291a6-7a72-48f0-bf8c-80dd9ed61138" (UID: "0d6291a6-7a72-48f0-bf8c-80dd9ed61138"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 11:16:13.135836 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.135669    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0d6291a6-7a72-48f0-bf8c-80dd9ed61138" (UID: "0d6291a6-7a72-48f0-bf8c-80dd9ed61138"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:16:13.136637 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.136609    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-kube-api-access-vg8nn" (OuterVolumeSpecName: "kube-api-access-vg8nn") pod "0d6291a6-7a72-48f0-bf8c-80dd9ed61138" (UID: "0d6291a6-7a72-48f0-bf8c-80dd9ed61138"). InnerVolumeSpecName "kube-api-access-vg8nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:16:13.225432 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.225409    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-credential-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:16:13.225432 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.225434    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg8nn\" (UniqueName: \"kubernetes.io/projected/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-kube-api-access-vg8nn\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:16:13.225572 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.225444    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-podinfo\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:16:13.225572 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.225453    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:16:13.225572 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.225461    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istio-envoy\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:16:13.225572 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.225468    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6291a6-7a72-48f0-bf8c-80dd9ed61138-istiod-ca-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:16:13.531268 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.531237    2561 generic.go:356] "Generic (PLEG): container finished" podID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerID="d5365beae84a8cacf12b797ae21a6e262ab392bcc83440205581305eaef7c4c1" exitCode=0
Sep 14 11:16:13.531428 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.531296    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"
Sep 14 11:16:13.531428 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.531317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" event={"ID":"0d6291a6-7a72-48f0-bf8c-80dd9ed61138","Type":"ContainerDied","Data":"d5365beae84a8cacf12b797ae21a6e262ab392bcc83440205581305eaef7c4c1"}
Sep 14 11:16:13.531428 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.531404    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw" event={"ID":"0d6291a6-7a72-48f0-bf8c-80dd9ed61138","Type":"ContainerDied","Data":"4a123b1fa5cd90a6611aa571c8ea20aedd0ebc4b8a1642cf3b396e839a25e37f"}
Sep 14 11:16:13.531533 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.531434    2561 scope.go:117] "RemoveContainer" containerID="d5365beae84a8cacf12b797ae21a6e262ab392bcc83440205581305eaef7c4c1"
Sep 14 11:16:13.541274 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.541257    2561 scope.go:117] "RemoveContainer" containerID="4522dcc307674370c5d536d58e1c68c8de1d0c9b47e810d362c886096f377d12"
Sep 14 11:16:13.551491 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.551473    2561 scope.go:117] "RemoveContainer" containerID="d5365beae84a8cacf12b797ae21a6e262ab392bcc83440205581305eaef7c4c1"
Sep 14 11:16:13.551791 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:16:13.551771    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5365beae84a8cacf12b797ae21a6e262ab392bcc83440205581305eaef7c4c1\": container with ID starting with d5365beae84a8cacf12b797ae21a6e262ab392bcc83440205581305eaef7c4c1 not found: ID does not exist" containerID="d5365beae84a8cacf12b797ae21a6e262ab392bcc83440205581305eaef7c4c1"
Sep 14 11:16:13.551866 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.551798    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5365beae84a8cacf12b797ae21a6e262ab392bcc83440205581305eaef7c4c1"} err="failed to get container status \"d5365beae84a8cacf12b797ae21a6e262ab392bcc83440205581305eaef7c4c1\": rpc error: code = NotFound desc = could not find container \"d5365beae84a8cacf12b797ae21a6e262ab392bcc83440205581305eaef7c4c1\": container with ID starting with d5365beae84a8cacf12b797ae21a6e262ab392bcc83440205581305eaef7c4c1 not found: ID does not exist"
Sep 14 11:16:13.551866 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.551815    2561 scope.go:117] "RemoveContainer" containerID="4522dcc307674370c5d536d58e1c68c8de1d0c9b47e810d362c886096f377d12"
Sep 14 11:16:13.552092 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:16:13.552075    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4522dcc307674370c5d536d58e1c68c8de1d0c9b47e810d362c886096f377d12\": container with ID starting with 4522dcc307674370c5d536d58e1c68c8de1d0c9b47e810d362c886096f377d12 not found: ID does not exist" containerID="4522dcc307674370c5d536d58e1c68c8de1d0c9b47e810d362c886096f377d12"
Sep 14 11:16:13.552138 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.552096    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4522dcc307674370c5d536d58e1c68c8de1d0c9b47e810d362c886096f377d12"} err="failed to get container status \"4522dcc307674370c5d536d58e1c68c8de1d0c9b47e810d362c886096f377d12\": rpc error: code = NotFound desc = could not find container \"4522dcc307674370c5d536d58e1c68c8de1d0c9b47e810d362c886096f377d12\": container with ID starting with 4522dcc307674370c5d536d58e1c68c8de1d0c9b47e810d362c886096f377d12 not found: ID does not exist"
Sep 14 11:16:13.575104 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.575084    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"]
Sep 14 11:16:13.582797 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.582772    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ebf73-openshift-default-ff4c898c4-dvlvw"]
Sep 14 11:16:13.804094 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:13.804017    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" path="/var/lib/kubelet/pods/0d6291a6-7a72-48f0-bf8c-80dd9ed61138/volumes"
Sep 14 11:16:15.516612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:15.516581    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:16:21.724123 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:21.724086    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:16:32.652671 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:32.652630    2561 generic.go:356] "Generic (PLEG): container finished" podID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerID="b3c0dae808ff4e224653e69b400c12902365cc917c32444dcbc78dc01e88e6a6" exitCode=0
Sep 14 11:16:32.653097 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:32.652702    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt" event={"ID":"947e8ad6-e7dc-4740-a0e7-7a05962f91f3","Type":"ContainerDied","Data":"b3c0dae808ff4e224653e69b400c12902365cc917c32444dcbc78dc01e88e6a6"}
Sep 14 11:16:32.653141 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:32.653102    2561 scope.go:117] "RemoveContainer" containerID="b3c0dae808ff4e224653e69b400c12902365cc917c32444dcbc78dc01e88e6a6"
Sep 14 11:16:32.654503 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:32.654469    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:16:32.654602 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:32.654585    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:16:32.654683 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:32.654631    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:16:33.665765 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:33.665700    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt" event={"ID":"947e8ad6-e7dc-4740-a0e7-7a05962f91f3","Type":"ContainerStarted","Data":"8d3ad8c54e500931f2368e6a1ca0e13ec44982d77d2fe65e1b42f07d727c5a61"}
Sep 14 11:16:34.560327 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:34.560282    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:16:34.560544 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:34.560413    2561 patch_prober.go:28] interesting pod/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 14 11:16:34.560544 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:34.560472    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt" podUID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 14 11:16:35.560171 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:35.560126    2561 patch_prober.go:28] interesting pod/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 14 11:16:35.560653 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:35.560193    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt" podUID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 14 11:16:36.559308 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:36.559268    2561 patch_prober.go:28] interesting pod/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 14 11:16:36.559612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:36.559579    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt" podUID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 14 11:16:37.559781 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:37.559739    2561 patch_prober.go:28] interesting pod/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 14 11:16:37.560303 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:37.559837    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt" podUID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 14 11:16:38.559818 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:38.559704    2561 patch_prober.go:28] interesting pod/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 14 11:16:38.559818 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:38.559786    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt" podUID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 14 11:16:39.563674 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:39.563643    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:16:39.564150 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:39.564096    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:16:39.564871 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:39.564853    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:16:43.417463 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:43.417426    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:16:53.112813 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.112775    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"]
Sep 14 11:16:53.113316 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.112974    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25" podUID="682102ab-c669-4efd-a4a6-b3fdd1c5ab5f" containerName="maas-api" containerID="cri-o://7c3bba8f1a3bb3d67d7122d66270344cdb8ff58e5242f65683299cafecbc9ef2" gracePeriod=30
Sep 14 11:16:53.386280 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.386256    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:16:53.521071 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.521042    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-openshift-service-ca\") pod \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") "
Sep 14 11:16:53.521238 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.521086    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-metrics-certs\") pod \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") "
Sep 14 11:16:53.521238 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.521155    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hhmd\" (UniqueName: \"kubernetes.io/projected/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-kube-api-access-7hhmd\") pod \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") "
Sep 14 11:16:53.521238 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.521173    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-maas-api-tls\") pod \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\" (UID: \"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f\") "
Sep 14 11:16:53.521484 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.521454    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "682102ab-c669-4efd-a4a6-b3fdd1c5ab5f" (UID: "682102ab-c669-4efd-a4a6-b3fdd1c5ab5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:16:53.523380 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.523347    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "682102ab-c669-4efd-a4a6-b3fdd1c5ab5f" (UID: "682102ab-c669-4efd-a4a6-b3fdd1c5ab5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:16:53.523485 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.523398    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-kube-api-access-7hhmd" (OuterVolumeSpecName: "kube-api-access-7hhmd") pod "682102ab-c669-4efd-a4a6-b3fdd1c5ab5f" (UID: "682102ab-c669-4efd-a4a6-b3fdd1c5ab5f"). InnerVolumeSpecName "kube-api-access-7hhmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:16:53.523485 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.523434    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "682102ab-c669-4efd-a4a6-b3fdd1c5ab5f" (UID: "682102ab-c669-4efd-a4a6-b3fdd1c5ab5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:16:53.622305 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.622192    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:16:53.622305 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.622220    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:16:53.622305 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.622230    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hhmd\" (UniqueName: \"kubernetes.io/projected/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-kube-api-access-7hhmd\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:16:53.622305 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.622239    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:16:53.792861 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.792828    2561 generic.go:356] "Generic (PLEG): container finished" podID="682102ab-c669-4efd-a4a6-b3fdd1c5ab5f" containerID="7c3bba8f1a3bb3d67d7122d66270344cdb8ff58e5242f65683299cafecbc9ef2" exitCode=0
Sep 14 11:16:53.793042 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.792875    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"
Sep 14 11:16:53.793042 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.792913    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25" event={"ID":"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f","Type":"ContainerDied","Data":"7c3bba8f1a3bb3d67d7122d66270344cdb8ff58e5242f65683299cafecbc9ef2"}
Sep 14 11:16:53.793042 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.792964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25" event={"ID":"682102ab-c669-4efd-a4a6-b3fdd1c5ab5f","Type":"ContainerDied","Data":"504b76eca52dd32f4d43277f8813425a7243156f6d2792b18c2338d21564bdd8"}
Sep 14 11:16:53.793042 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.792989    2561 scope.go:117] "RemoveContainer" containerID="7c3bba8f1a3bb3d67d7122d66270344cdb8ff58e5242f65683299cafecbc9ef2"
Sep 14 11:16:53.806651 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.806604    2561 scope.go:117] "RemoveContainer" containerID="7c3bba8f1a3bb3d67d7122d66270344cdb8ff58e5242f65683299cafecbc9ef2"
Sep 14 11:16:53.807206 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:16:53.807175    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3bba8f1a3bb3d67d7122d66270344cdb8ff58e5242f65683299cafecbc9ef2\": container with ID starting with 7c3bba8f1a3bb3d67d7122d66270344cdb8ff58e5242f65683299cafecbc9ef2 not found: ID does not exist" containerID="7c3bba8f1a3bb3d67d7122d66270344cdb8ff58e5242f65683299cafecbc9ef2"
Sep 14 11:16:53.807303 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.807216    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3bba8f1a3bb3d67d7122d66270344cdb8ff58e5242f65683299cafecbc9ef2"} err="failed to get container status \"7c3bba8f1a3bb3d67d7122d66270344cdb8ff58e5242f65683299cafecbc9ef2\": rpc error: code = NotFound desc = could not find container \"7c3bba8f1a3bb3d67d7122d66270344cdb8ff58e5242f65683299cafecbc9ef2\": container with ID starting with 7c3bba8f1a3bb3d67d7122d66270344cdb8ff58e5242f65683299cafecbc9ef2 not found: ID does not exist"
Sep 14 11:16:53.827073 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.827048    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"]
Sep 14 11:16:53.831905 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:53.831884    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29e74e-79994f4d9f-qvs25"]
Sep 14 11:16:55.803651 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:55.803620    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="682102ab-c669-4efd-a4a6-b3fdd1c5ab5f" path="/var/lib/kubelet/pods/682102ab-c669-4efd-a4a6-b3fdd1c5ab5f/volumes"
Sep 14 11:16:57.614835 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:16:57.614799    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:17:14.748305 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:14.748272    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:17:17.087287 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.087254    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"]
Sep 14 11:17:17.087734 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.087497    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49" podUID="dc231966-027b-4833-aba3-1db6f05da322" containerName="maas-api" containerID="cri-o://9fd930715b592d1763508640db59a09c20b77467aa7b7975d9e25cfd77355715" gracePeriod=30
Sep 14 11:17:17.345409 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.345343    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:17:17.446136 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.446092    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twl5c\" (UniqueName: \"kubernetes.io/projected/dc231966-027b-4833-aba3-1db6f05da322-kube-api-access-twl5c\") pod \"dc231966-027b-4833-aba3-1db6f05da322\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") "
Sep 14 11:17:17.446343 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.446229    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-metrics-certs\") pod \"dc231966-027b-4833-aba3-1db6f05da322\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") "
Sep 14 11:17:17.446343 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.446290    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc231966-027b-4833-aba3-1db6f05da322-openshift-service-ca\") pod \"dc231966-027b-4833-aba3-1db6f05da322\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") "
Sep 14 11:17:17.446453 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.446402    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-maas-api-tls\") pod \"dc231966-027b-4833-aba3-1db6f05da322\" (UID: \"dc231966-027b-4833-aba3-1db6f05da322\") "
Sep 14 11:17:17.446825 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.446795    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc231966-027b-4833-aba3-1db6f05da322-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc231966-027b-4833-aba3-1db6f05da322" (UID: "dc231966-027b-4833-aba3-1db6f05da322"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:17:17.448630 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.448593    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc231966-027b-4833-aba3-1db6f05da322" (UID: "dc231966-027b-4833-aba3-1db6f05da322"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:17:17.449045 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.449016    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc231966-027b-4833-aba3-1db6f05da322-kube-api-access-twl5c" (OuterVolumeSpecName: "kube-api-access-twl5c") pod "dc231966-027b-4833-aba3-1db6f05da322" (UID: "dc231966-027b-4833-aba3-1db6f05da322"). InnerVolumeSpecName "kube-api-access-twl5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:17:17.449165 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.449141    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dc231966-027b-4833-aba3-1db6f05da322" (UID: "dc231966-027b-4833-aba3-1db6f05da322"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:17:17.547294 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.547256    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twl5c\" (UniqueName: \"kubernetes.io/projected/dc231966-027b-4833-aba3-1db6f05da322-kube-api-access-twl5c\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:17.547294 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.547292    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:17.547294 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.547301    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc231966-027b-4833-aba3-1db6f05da322-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:17.547541 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.547311    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc231966-027b-4833-aba3-1db6f05da322-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:17.663692 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.663656    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7845-6a81f9"]
Sep 14 11:17:17.664173 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.664156    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="682102ab-c669-4efd-a4a6-b3fdd1c5ab5f" containerName="maas-api"
Sep 14 11:17:17.664250 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.664174    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="682102ab-c669-4efd-a4a6-b3fdd1c5ab5f" containerName="maas-api"
Sep 14 11:17:17.664250 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.664195    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc231966-027b-4833-aba3-1db6f05da322" containerName="maas-api"
Sep 14 11:17:17.664250 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.664201    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc231966-027b-4833-aba3-1db6f05da322" containerName="maas-api"
Sep 14 11:17:17.664250 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.664214    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy"
Sep 14 11:17:17.664250 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.664219    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy"
Sep 14 11:17:17.664250 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.664229    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy"
Sep 14 11:17:17.664250 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.664234    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy"
Sep 14 11:17:17.664591 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.664298    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc231966-027b-4833-aba3-1db6f05da322" containerName="maas-api"
Sep 14 11:17:17.664591 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.664307    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy"
Sep 14 11:17:17.664591 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.664313    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d6291a6-7a72-48f0-bf8c-80dd9ed61138" containerName="istio-proxy"
Sep 14 11:17:17.664591 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.664321    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="682102ab-c669-4efd-a4a6-b3fdd1c5ab5f" containerName="maas-api"
Sep 14 11:17:17.667272 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.667257    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7845-6a81f9"
Sep 14 11:17:17.670117 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.670095    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-stw9c\""
Sep 14 11:17:17.674518 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.674497    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7845-6a81f9"]
Sep 14 11:17:17.749195 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.749168    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbrwn\" (UniqueName: \"kubernetes.io/projected/b7ed0e43-21e3-40ba-9759-14bd19b53cef-kube-api-access-gbrwn\") pod \"test-curl-7845-6a81f9\" (UID: \"b7ed0e43-21e3-40ba-9759-14bd19b53cef\") " pod="opendatahub/test-curl-7845-6a81f9"
Sep 14 11:17:17.849888 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.849862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbrwn\" (UniqueName: \"kubernetes.io/projected/b7ed0e43-21e3-40ba-9759-14bd19b53cef-kube-api-access-gbrwn\") pod \"test-curl-7845-6a81f9\" (UID: \"b7ed0e43-21e3-40ba-9759-14bd19b53cef\") " pod="opendatahub/test-curl-7845-6a81f9"
Sep 14 11:17:17.859387 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.859359    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbrwn\" (UniqueName: \"kubernetes.io/projected/b7ed0e43-21e3-40ba-9759-14bd19b53cef-kube-api-access-gbrwn\") pod \"test-curl-7845-6a81f9\" (UID: \"b7ed0e43-21e3-40ba-9759-14bd19b53cef\") " pod="opendatahub/test-curl-7845-6a81f9"
Sep 14 11:17:17.939755 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.939659    2561 generic.go:356] "Generic (PLEG): container finished" podID="dc231966-027b-4833-aba3-1db6f05da322" containerID="9fd930715b592d1763508640db59a09c20b77467aa7b7975d9e25cfd77355715" exitCode=0
Sep 14 11:17:17.939755 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.939691    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49" event={"ID":"dc231966-027b-4833-aba3-1db6f05da322","Type":"ContainerDied","Data":"9fd930715b592d1763508640db59a09c20b77467aa7b7975d9e25cfd77355715"}
Sep 14 11:17:17.939755 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.939753    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"
Sep 14 11:17:17.939970 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.939773    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49" event={"ID":"dc231966-027b-4833-aba3-1db6f05da322","Type":"ContainerDied","Data":"e9597fb29a4f6a9aa0baf5a57b5c84f89af3114bd7cd93bc27d6c0fa668d720a"}
Sep 14 11:17:17.939970 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.939805    2561 scope.go:117] "RemoveContainer" containerID="9fd930715b592d1763508640db59a09c20b77467aa7b7975d9e25cfd77355715"
Sep 14 11:17:17.951119 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.951093    2561 scope.go:117] "RemoveContainer" containerID="9fd930715b592d1763508640db59a09c20b77467aa7b7975d9e25cfd77355715"
Sep 14 11:17:17.951427 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:17:17.951403    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd930715b592d1763508640db59a09c20b77467aa7b7975d9e25cfd77355715\": container with ID starting with 9fd930715b592d1763508640db59a09c20b77467aa7b7975d9e25cfd77355715 not found: ID does not exist" containerID="9fd930715b592d1763508640db59a09c20b77467aa7b7975d9e25cfd77355715"
Sep 14 11:17:17.951512 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.951442    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd930715b592d1763508640db59a09c20b77467aa7b7975d9e25cfd77355715"} err="failed to get container status \"9fd930715b592d1763508640db59a09c20b77467aa7b7975d9e25cfd77355715\": rpc error: code = NotFound desc = could not find container \"9fd930715b592d1763508640db59a09c20b77467aa7b7975d9e25cfd77355715\": container with ID starting with 9fd930715b592d1763508640db59a09c20b77467aa7b7975d9e25cfd77355715 not found: ID does not exist"
Sep 14 11:17:17.963239 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.963211    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"]
Sep 14 11:17:17.970694 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.970672    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f8e582-6bdc869d58-4nq49"]
Sep 14 11:17:17.978357 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:17.978340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7845-6a81f9"
Sep 14 11:17:18.255316 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:18.255285    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7845-6a81f9"]
Sep 14 11:17:18.256721 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:17:18.256683    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7ed0e43_21e3_40ba_9759_14bd19b53cef.slice/crio-4ebfde0a561cdc102b8e75e679842db9a679d8b03021f67549f498ef6c0c29a5 WatchSource:0}: Error finding container 4ebfde0a561cdc102b8e75e679842db9a679d8b03021f67549f498ef6c0c29a5: Status 404 returned error can't find the container with id 4ebfde0a561cdc102b8e75e679842db9a679d8b03021f67549f498ef6c0c29a5
Sep 14 11:17:18.947863 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:18.947819    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7845-6a81f9" event={"ID":"b7ed0e43-21e3-40ba-9759-14bd19b53cef","Type":"ContainerStarted","Data":"75918c2ae1c82ed856802bdf60c56bc7f79fcbbbbef2081212a957ddcebd1935"}
Sep 14 11:17:18.947863 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:18.947870    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7845-6a81f9" event={"ID":"b7ed0e43-21e3-40ba-9759-14bd19b53cef","Type":"ContainerStarted","Data":"4ebfde0a561cdc102b8e75e679842db9a679d8b03021f67549f498ef6c0c29a5"}
Sep 14 11:17:18.964833 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:18.964785    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7845-6a81f9" podStartSLOduration=1.964768056 podStartE2EDuration="1.964768056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:17:18.963041775 +0000 UTC m=+1743.647726163" watchObservedRunningTime="2026-09-14 11:17:18.964768056 +0000 UTC m=+1743.649452445"
Sep 14 11:17:19.568961 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:19.568922    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7845-6a81f9"]
Sep 14 11:17:19.571684 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:19.571659    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7845-6a81f9"]
Sep 14 11:17:19.804348 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:19.804315    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc231966-027b-4833-aba3-1db6f05da322" path="/var/lib/kubelet/pods/dc231966-027b-4833-aba3-1db6f05da322/volumes"
Sep 14 11:17:19.955104 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:17:19.955073    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7845-6a81f9" container="test-curl-7845-6a81f9"
Sep 14 11:17:20.960405 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:20.960369    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7845-6a81f9" podUID="b7ed0e43-21e3-40ba-9759-14bd19b53cef" containerName="test-curl-7845-6a81f9" containerID="cri-o://75918c2ae1c82ed856802bdf60c56bc7f79fcbbbbef2081212a957ddcebd1935" gracePeriod=2
Sep 14 11:17:23.310878 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:23.310854    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7845-6a81f9"
Sep 14 11:17:23.402260 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:23.402235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbrwn\" (UniqueName: \"kubernetes.io/projected/b7ed0e43-21e3-40ba-9759-14bd19b53cef-kube-api-access-gbrwn\") pod \"b7ed0e43-21e3-40ba-9759-14bd19b53cef\" (UID: \"b7ed0e43-21e3-40ba-9759-14bd19b53cef\") "
Sep 14 11:17:23.404184 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:23.404160    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ed0e43-21e3-40ba-9759-14bd19b53cef-kube-api-access-gbrwn" (OuterVolumeSpecName: "kube-api-access-gbrwn") pod "b7ed0e43-21e3-40ba-9759-14bd19b53cef" (UID: "b7ed0e43-21e3-40ba-9759-14bd19b53cef"). InnerVolumeSpecName "kube-api-access-gbrwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:17:23.502889 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:23.502833    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbrwn\" (UniqueName: \"kubernetes.io/projected/b7ed0e43-21e3-40ba-9759-14bd19b53cef-kube-api-access-gbrwn\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:23.803956 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:23.803879    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7ed0e43-21e3-40ba-9759-14bd19b53cef" path="/var/lib/kubelet/pods/b7ed0e43-21e3-40ba-9759-14bd19b53cef/volumes"
Sep 14 11:17:23.979276 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:23.979239    2561 generic.go:356] "Generic (PLEG): container finished" podID="b7ed0e43-21e3-40ba-9759-14bd19b53cef" containerID="75918c2ae1c82ed856802bdf60c56bc7f79fcbbbbef2081212a957ddcebd1935" exitCode=137
Sep 14 11:17:23.979515 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:23.979281    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7845-6a81f9"
Sep 14 11:17:23.979515 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:23.979338    2561 scope.go:117] "RemoveContainer" containerID="75918c2ae1c82ed856802bdf60c56bc7f79fcbbbbef2081212a957ddcebd1935"
Sep 14 11:17:23.988876 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:23.988857    2561 scope.go:117] "RemoveContainer" containerID="75918c2ae1c82ed856802bdf60c56bc7f79fcbbbbef2081212a957ddcebd1935"
Sep 14 11:17:23.989147 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:17:23.989130    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75918c2ae1c82ed856802bdf60c56bc7f79fcbbbbef2081212a957ddcebd1935\": container with ID starting with 75918c2ae1c82ed856802bdf60c56bc7f79fcbbbbef2081212a957ddcebd1935 not found: ID does not exist" containerID="75918c2ae1c82ed856802bdf60c56bc7f79fcbbbbef2081212a957ddcebd1935"
Sep 14 11:17:23.989206 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:23.989154    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75918c2ae1c82ed856802bdf60c56bc7f79fcbbbbef2081212a957ddcebd1935"} err="failed to get container status \"75918c2ae1c82ed856802bdf60c56bc7f79fcbbbbef2081212a957ddcebd1935\": rpc error: code = NotFound desc = could not find container \"75918c2ae1c82ed856802bdf60c56bc7f79fcbbbbef2081212a957ddcebd1935\": container with ID starting with 75918c2ae1c82ed856802bdf60c56bc7f79fcbbbbef2081212a957ddcebd1935 not found: ID does not exist"
Sep 14 11:17:29.066838 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:29.066806    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"]
Sep 14 11:17:29.067207 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:29.067052    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt" podUID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerName="istio-proxy" containerID="cri-o://8d3ad8c54e500931f2368e6a1ca0e13ec44982d77d2fe65e1b42f07d727c5a61" gracePeriod=30
Sep 14 11:17:34.316762 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.316734    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:17:34.410690 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.410660    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-workload-socket\") pod \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") "
Sep 14 11:17:34.410893 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.410754    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istiod-ca-cert\") pod \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") "
Sep 14 11:17:34.410893 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.410806    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-credential-socket\") pod \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") "
Sep 14 11:17:34.410893 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.410831    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz4dt\" (UniqueName: \"kubernetes.io/projected/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-kube-api-access-qz4dt\") pod \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") "
Sep 14 11:17:34.410893 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.410886    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-envoy\") pod \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") "
Sep 14 11:17:34.411200 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.410909    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-workload-certs\") pod \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") "
Sep 14 11:17:34.411200 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.410969    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "947e8ad6-e7dc-4740-a0e7-7a05962f91f3" (UID: "947e8ad6-e7dc-4740-a0e7-7a05962f91f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:17:34.411200 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.411007    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-podinfo\") pod \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") "
Sep 14 11:17:34.411200 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.411062    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-data\") pod \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") "
Sep 14 11:17:34.411200 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.411064    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "947e8ad6-e7dc-4740-a0e7-7a05962f91f3" (UID: "947e8ad6-e7dc-4740-a0e7-7a05962f91f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:17:34.411200 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.411095    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-token\") pod \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\" (UID: \"947e8ad6-e7dc-4740-a0e7-7a05962f91f3\") "
Sep 14 11:17:34.411200 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.411159    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "947e8ad6-e7dc-4740-a0e7-7a05962f91f3" (UID: "947e8ad6-e7dc-4740-a0e7-7a05962f91f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:17:34.412050 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.411331    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "947e8ad6-e7dc-4740-a0e7-7a05962f91f3" (UID: "947e8ad6-e7dc-4740-a0e7-7a05962f91f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:17:34.412050 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.411356    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "947e8ad6-e7dc-4740-a0e7-7a05962f91f3" (UID: "947e8ad6-e7dc-4740-a0e7-7a05962f91f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:17:34.412050 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.411473    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-workload-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:34.412050 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.411493    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istiod-ca-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:34.412050 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.411507    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-credential-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:34.412050 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.411520    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-workload-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:34.412050 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.411535    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-data\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:34.413218 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.413186    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "947e8ad6-e7dc-4740-a0e7-7a05962f91f3" (UID: "947e8ad6-e7dc-4740-a0e7-7a05962f91f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:17:34.413341 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.413314    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "947e8ad6-e7dc-4740-a0e7-7a05962f91f3" (UID: "947e8ad6-e7dc-4740-a0e7-7a05962f91f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 11:17:34.413341 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.413336    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-kube-api-access-qz4dt" (OuterVolumeSpecName: "kube-api-access-qz4dt") pod "947e8ad6-e7dc-4740-a0e7-7a05962f91f3" (UID: "947e8ad6-e7dc-4740-a0e7-7a05962f91f3"). InnerVolumeSpecName "kube-api-access-qz4dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:17:34.413486 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.413369    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "947e8ad6-e7dc-4740-a0e7-7a05962f91f3" (UID: "947e8ad6-e7dc-4740-a0e7-7a05962f91f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:17:34.512669 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.512639    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz4dt\" (UniqueName: \"kubernetes.io/projected/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-kube-api-access-qz4dt\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:34.512669 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.512666    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-envoy\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:34.512669 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.512675    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-podinfo\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:34.512916 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.512684    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/947e8ad6-e7dc-4740-a0e7-7a05962f91f3-istio-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:34.649210 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:34.649172    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:17:35.050267 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:35.050231    2561 generic.go:356] "Generic (PLEG): container finished" podID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerID="8d3ad8c54e500931f2368e6a1ca0e13ec44982d77d2fe65e1b42f07d727c5a61" exitCode=0
Sep 14 11:17:35.050454 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:35.050300    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"
Sep 14 11:17:35.050454 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:35.050325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt" event={"ID":"947e8ad6-e7dc-4740-a0e7-7a05962f91f3","Type":"ContainerDied","Data":"8d3ad8c54e500931f2368e6a1ca0e13ec44982d77d2fe65e1b42f07d727c5a61"}
Sep 14 11:17:35.050454 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:35.050400    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt" event={"ID":"947e8ad6-e7dc-4740-a0e7-7a05962f91f3","Type":"ContainerDied","Data":"2e428bcd6d3968f83af6559b591c23ad289d67131df363d372fb318f425dc4b9"}
Sep 14 11:17:35.050454 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:35.050430    2561 scope.go:117] "RemoveContainer" containerID="8d3ad8c54e500931f2368e6a1ca0e13ec44982d77d2fe65e1b42f07d727c5a61"
Sep 14 11:17:35.061483 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:35.061464    2561 scope.go:117] "RemoveContainer" containerID="b3c0dae808ff4e224653e69b400c12902365cc917c32444dcbc78dc01e88e6a6"
Sep 14 11:17:35.069918 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:35.069901    2561 scope.go:117] "RemoveContainer" containerID="8d3ad8c54e500931f2368e6a1ca0e13ec44982d77d2fe65e1b42f07d727c5a61"
Sep 14 11:17:35.070194 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:17:35.070175    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d3ad8c54e500931f2368e6a1ca0e13ec44982d77d2fe65e1b42f07d727c5a61\": container with ID starting with 8d3ad8c54e500931f2368e6a1ca0e13ec44982d77d2fe65e1b42f07d727c5a61 not found: ID does not exist" containerID="8d3ad8c54e500931f2368e6a1ca0e13ec44982d77d2fe65e1b42f07d727c5a61"
Sep 14 11:17:35.070250 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:35.070203    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d3ad8c54e500931f2368e6a1ca0e13ec44982d77d2fe65e1b42f07d727c5a61"} err="failed to get container status \"8d3ad8c54e500931f2368e6a1ca0e13ec44982d77d2fe65e1b42f07d727c5a61\": rpc error: code = NotFound desc = could not find container \"8d3ad8c54e500931f2368e6a1ca0e13ec44982d77d2fe65e1b42f07d727c5a61\": container with ID starting with 8d3ad8c54e500931f2368e6a1ca0e13ec44982d77d2fe65e1b42f07d727c5a61 not found: ID does not exist"
Sep 14 11:17:35.070250 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:35.070220    2561 scope.go:117] "RemoveContainer" containerID="b3c0dae808ff4e224653e69b400c12902365cc917c32444dcbc78dc01e88e6a6"
Sep 14 11:17:35.070445 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:17:35.070429    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c0dae808ff4e224653e69b400c12902365cc917c32444dcbc78dc01e88e6a6\": container with ID starting with b3c0dae808ff4e224653e69b400c12902365cc917c32444dcbc78dc01e88e6a6 not found: ID does not exist" containerID="b3c0dae808ff4e224653e69b400c12902365cc917c32444dcbc78dc01e88e6a6"
Sep 14 11:17:35.070488 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:35.070448    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c0dae808ff4e224653e69b400c12902365cc917c32444dcbc78dc01e88e6a6"} err="failed to get container status \"b3c0dae808ff4e224653e69b400c12902365cc917c32444dcbc78dc01e88e6a6\": rpc error: code = NotFound desc = could not find container \"b3c0dae808ff4e224653e69b400c12902365cc917c32444dcbc78dc01e88e6a6\": container with ID starting with b3c0dae808ff4e224653e69b400c12902365cc917c32444dcbc78dc01e88e6a6 not found: ID does not exist"
Sep 14 11:17:35.082529 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:35.082510    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"]
Sep 14 11:17:35.088126 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:35.088102    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f8e582-openshift-default-57b8c8cc8-rfrnt"]
Sep 14 11:17:35.804964 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:35.804933    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" path="/var/lib/kubelet/pods/947e8ad6-e7dc-4740-a0e7-7a05962f91f3/volumes"
Sep 14 11:17:40.205227 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.205191    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"]
Sep 14 11:17:40.205744 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.205392    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj" podUID="67039613-681d-44e6-b719-ba5086dd7823" containerName="maas-api" containerID="cri-o://bb27bb8554d6c91d0fd03c5dab0c2e222371c06e9517bd9a8bb984bc69a160ab" gracePeriod=30
Sep 14 11:17:40.255062 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.255028    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm"]
Sep 14 11:17:40.262386 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.262360    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ae9d47-29823075-gx4cm"]
Sep 14 11:17:40.460447 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.460361    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:17:40.571781 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.571747    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67039613-681d-44e6-b719-ba5086dd7823-openshift-service-ca\") pod \"67039613-681d-44e6-b719-ba5086dd7823\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") "
Sep 14 11:17:40.571935 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.571797    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-metrics-certs\") pod \"67039613-681d-44e6-b719-ba5086dd7823\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") "
Sep 14 11:17:40.571935 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.571846    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hmlg\" (UniqueName: \"kubernetes.io/projected/67039613-681d-44e6-b719-ba5086dd7823-kube-api-access-7hmlg\") pod \"67039613-681d-44e6-b719-ba5086dd7823\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") "
Sep 14 11:17:40.571935 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.571929    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-maas-api-tls\") pod \"67039613-681d-44e6-b719-ba5086dd7823\" (UID: \"67039613-681d-44e6-b719-ba5086dd7823\") "
Sep 14 11:17:40.572162 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.572128    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67039613-681d-44e6-b719-ba5086dd7823-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67039613-681d-44e6-b719-ba5086dd7823" (UID: "67039613-681d-44e6-b719-ba5086dd7823"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:17:40.573908 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.573884    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "67039613-681d-44e6-b719-ba5086dd7823" (UID: "67039613-681d-44e6-b719-ba5086dd7823"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:17:40.573998 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.573908    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67039613-681d-44e6-b719-ba5086dd7823-kube-api-access-7hmlg" (OuterVolumeSpecName: "kube-api-access-7hmlg") pod "67039613-681d-44e6-b719-ba5086dd7823" (UID: "67039613-681d-44e6-b719-ba5086dd7823"). InnerVolumeSpecName "kube-api-access-7hmlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:17:40.573998 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.573989    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67039613-681d-44e6-b719-ba5086dd7823" (UID: "67039613-681d-44e6-b719-ba5086dd7823"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:17:40.672639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.672610    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:40.672639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.672636    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67039613-681d-44e6-b719-ba5086dd7823-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:40.672639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.672646    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67039613-681d-44e6-b719-ba5086dd7823-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:40.672935 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:40.672655    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hmlg\" (UniqueName: \"kubernetes.io/projected/67039613-681d-44e6-b719-ba5086dd7823-kube-api-access-7hmlg\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:17:41.090308 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:41.090272    2561 generic.go:356] "Generic (PLEG): container finished" podID="67039613-681d-44e6-b719-ba5086dd7823" containerID="bb27bb8554d6c91d0fd03c5dab0c2e222371c06e9517bd9a8bb984bc69a160ab" exitCode=0
Sep 14 11:17:41.090471 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:41.090321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj" event={"ID":"67039613-681d-44e6-b719-ba5086dd7823","Type":"ContainerDied","Data":"bb27bb8554d6c91d0fd03c5dab0c2e222371c06e9517bd9a8bb984bc69a160ab"}
Sep 14 11:17:41.090471 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:41.090363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj" event={"ID":"67039613-681d-44e6-b719-ba5086dd7823","Type":"ContainerDied","Data":"b6ece2cbb2b7efb10c36fc610e485fd53d43d0d5a330944c201064c792ae4ac1"}
Sep 14 11:17:41.090471 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:41.090386    2561 scope.go:117] "RemoveContainer" containerID="bb27bb8554d6c91d0fd03c5dab0c2e222371c06e9517bd9a8bb984bc69a160ab"
Sep 14 11:17:41.090471 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:41.090324    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"
Sep 14 11:17:41.100434 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:41.100414    2561 scope.go:117] "RemoveContainer" containerID="bb27bb8554d6c91d0fd03c5dab0c2e222371c06e9517bd9a8bb984bc69a160ab"
Sep 14 11:17:41.100728 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:17:41.100695    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb27bb8554d6c91d0fd03c5dab0c2e222371c06e9517bd9a8bb984bc69a160ab\": container with ID starting with bb27bb8554d6c91d0fd03c5dab0c2e222371c06e9517bd9a8bb984bc69a160ab not found: ID does not exist" containerID="bb27bb8554d6c91d0fd03c5dab0c2e222371c06e9517bd9a8bb984bc69a160ab"
Sep 14 11:17:41.100808 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:41.100736    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb27bb8554d6c91d0fd03c5dab0c2e222371c06e9517bd9a8bb984bc69a160ab"} err="failed to get container status \"bb27bb8554d6c91d0fd03c5dab0c2e222371c06e9517bd9a8bb984bc69a160ab\": rpc error: code = NotFound desc = could not find container \"bb27bb8554d6c91d0fd03c5dab0c2e222371c06e9517bd9a8bb984bc69a160ab\": container with ID starting with bb27bb8554d6c91d0fd03c5dab0c2e222371c06e9517bd9a8bb984bc69a160ab not found: ID does not exist"
Sep 14 11:17:41.127182 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:41.127156    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"]
Sep 14 11:17:41.140038 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:41.140016    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ae9d47-699f6d7d54-42gmj"]
Sep 14 11:17:41.803102 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:41.803072    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67039613-681d-44e6-b719-ba5086dd7823" path="/var/lib/kubelet/pods/67039613-681d-44e6-b719-ba5086dd7823/volumes"
Sep 14 11:17:41.803453 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:41.803395    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb12f249-790a-45d1-a2bb-c8a8103c8131" path="/var/lib/kubelet/pods/eb12f249-790a-45d1-a2bb-c8a8103c8131/volumes"
Sep 14 11:17:45.213465 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:45.213431    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:17:59.795333 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:59.795300    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"]
Sep 14 11:17:59.795837 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:17:59.795525    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp" podUID="5f21905f-da93-4a4b-91e4-cabfe6cb9bf0" containerName="maas-api" containerID="cri-o://ebfa76361b84166c34b3e113356164a364ca464e764133def3dd04fb66c18a7c" gracePeriod=30
Sep 14 11:18:00.052251 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.052184    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:18:00.153019 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.152982    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-maas-api-tls\") pod \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") "
Sep 14 11:18:00.153306 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.153035    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-openshift-service-ca\") pod \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") "
Sep 14 11:18:00.153306 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.153053    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-metrics-certs\") pod \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") "
Sep 14 11:18:00.153306 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.153123    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vld5j\" (UniqueName: \"kubernetes.io/projected/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-kube-api-access-vld5j\") pod \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\" (UID: \"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0\") "
Sep 14 11:18:00.153501 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.153433    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f21905f-da93-4a4b-91e4-cabfe6cb9bf0" (UID: "5f21905f-da93-4a4b-91e4-cabfe6cb9bf0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:18:00.155520 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.155474    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5f21905f-da93-4a4b-91e4-cabfe6cb9bf0" (UID: "5f21905f-da93-4a4b-91e4-cabfe6cb9bf0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:18:00.155686 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.155531    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-kube-api-access-vld5j" (OuterVolumeSpecName: "kube-api-access-vld5j") pod "5f21905f-da93-4a4b-91e4-cabfe6cb9bf0" (UID: "5f21905f-da93-4a4b-91e4-cabfe6cb9bf0"). InnerVolumeSpecName "kube-api-access-vld5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:18:00.155686 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.155626    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f21905f-da93-4a4b-91e4-cabfe6cb9bf0" (UID: "5f21905f-da93-4a4b-91e4-cabfe6cb9bf0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:18:00.209011 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.208970    2561 generic.go:356] "Generic (PLEG): container finished" podID="5f21905f-da93-4a4b-91e4-cabfe6cb9bf0" containerID="ebfa76361b84166c34b3e113356164a364ca464e764133def3dd04fb66c18a7c" exitCode=0
Sep 14 11:18:00.209206 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.209016    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"
Sep 14 11:18:00.209206 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.209055    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp" event={"ID":"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0","Type":"ContainerDied","Data":"ebfa76361b84166c34b3e113356164a364ca464e764133def3dd04fb66c18a7c"}
Sep 14 11:18:00.209206 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.209123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp" event={"ID":"5f21905f-da93-4a4b-91e4-cabfe6cb9bf0","Type":"ContainerDied","Data":"29eccbf60f8fdaa4d7044a04af42deb1a979ec39b82cb58753692a9be59fc639"}
Sep 14 11:18:00.209206 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.209154    2561 scope.go:117] "RemoveContainer" containerID="ebfa76361b84166c34b3e113356164a364ca464e764133def3dd04fb66c18a7c"
Sep 14 11:18:00.222556 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.222368    2561 scope.go:117] "RemoveContainer" containerID="ebfa76361b84166c34b3e113356164a364ca464e764133def3dd04fb66c18a7c"
Sep 14 11:18:00.222883 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:18:00.222846    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfa76361b84166c34b3e113356164a364ca464e764133def3dd04fb66c18a7c\": container with ID starting with ebfa76361b84166c34b3e113356164a364ca464e764133def3dd04fb66c18a7c not found: ID does not exist" containerID="ebfa76361b84166c34b3e113356164a364ca464e764133def3dd04fb66c18a7c"
Sep 14 11:18:00.223035 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.222895    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfa76361b84166c34b3e113356164a364ca464e764133def3dd04fb66c18a7c"} err="failed to get container status \"ebfa76361b84166c34b3e113356164a364ca464e764133def3dd04fb66c18a7c\": rpc error: code = NotFound desc = could not find container \"ebfa76361b84166c34b3e113356164a364ca464e764133def3dd04fb66c18a7c\": container with ID starting with ebfa76361b84166c34b3e113356164a364ca464e764133def3dd04fb66c18a7c not found: ID does not exist"
Sep 14 11:18:00.252672 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.252637    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"]
Sep 14 11:18:00.254848 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.254822    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:00.254848 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.254850    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:00.255083 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.254866    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:00.255083 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.254881    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vld5j\" (UniqueName: \"kubernetes.io/projected/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0-kube-api-access-vld5j\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:00.269075 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:00.269047    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-22b014-55b8d7fcc8-vv4hp"]
Sep 14 11:18:01.803729 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:01.803693    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f21905f-da93-4a4b-91e4-cabfe6cb9bf0" path="/var/lib/kubelet/pods/5f21905f-da93-4a4b-91e4-cabfe6cb9bf0/volumes"
Sep 14 11:18:03.391354 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:03.391321    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:18:07.555227 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:07.555195    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"]
Sep 14 11:18:07.555823 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:07.555392    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv" podUID="9e419258-d263-4df1-aaf0-422eace67cc9" containerName="istio-proxy" containerID="cri-o://02d9283172b7b2ba2a7ab2d8304070e06ad3231d00c0191462787726cca96a55" gracePeriod=30
Sep 14 11:18:12.809825 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.809802    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:18:12.867515 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.867491    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-workload-socket\") pod \"9e419258-d263-4df1-aaf0-422eace67cc9\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") "
Sep 14 11:18:12.867692 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.867523    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-workload-certs\") pod \"9e419258-d263-4df1-aaf0-422eace67cc9\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") "
Sep 14 11:18:12.867692 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.867558    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e419258-d263-4df1-aaf0-422eace67cc9-istio-token\") pod \"9e419258-d263-4df1-aaf0-422eace67cc9\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") "
Sep 14 11:18:12.867692 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.867610    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-credential-socket\") pod \"9e419258-d263-4df1-aaf0-422eace67cc9\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") "
Sep 14 11:18:12.867692 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.867638    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvwsm\" (UniqueName: \"kubernetes.io/projected/9e419258-d263-4df1-aaf0-422eace67cc9-kube-api-access-cvwsm\") pod \"9e419258-d263-4df1-aaf0-422eace67cc9\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") "
Sep 14 11:18:12.867692 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.867664    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-istio-envoy\") pod \"9e419258-d263-4df1-aaf0-422eace67cc9\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") "
Sep 14 11:18:12.867692 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.867688    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e419258-d263-4df1-aaf0-422eace67cc9-istio-podinfo\") pod \"9e419258-d263-4df1-aaf0-422eace67cc9\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") "
Sep 14 11:18:12.868097 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.867746    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e419258-d263-4df1-aaf0-422eace67cc9-istiod-ca-cert\") pod \"9e419258-d263-4df1-aaf0-422eace67cc9\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") "
Sep 14 11:18:12.868097 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.867779    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-istio-data\") pod \"9e419258-d263-4df1-aaf0-422eace67cc9\" (UID: \"9e419258-d263-4df1-aaf0-422eace67cc9\") "
Sep 14 11:18:12.868097 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.867854    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e419258-d263-4df1-aaf0-422eace67cc9" (UID: "9e419258-d263-4df1-aaf0-422eace67cc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:18:12.868097 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.867868    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e419258-d263-4df1-aaf0-422eace67cc9" (UID: "9e419258-d263-4df1-aaf0-422eace67cc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:18:12.868097 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.867979    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e419258-d263-4df1-aaf0-422eace67cc9" (UID: "9e419258-d263-4df1-aaf0-422eace67cc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:18:12.868376 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.868206    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e419258-d263-4df1-aaf0-422eace67cc9" (UID: "9e419258-d263-4df1-aaf0-422eace67cc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:18:12.868376 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.868291    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-workload-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:12.868376 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.868312    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-workload-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:12.868376 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.868325    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-credential-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:12.868376 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.868339    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-istio-data\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:12.868622 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.868445    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e419258-d263-4df1-aaf0-422eace67cc9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e419258-d263-4df1-aaf0-422eace67cc9" (UID: "9e419258-d263-4df1-aaf0-422eace67cc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:18:12.869988 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.869963    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e419258-d263-4df1-aaf0-422eace67cc9" (UID: "9e419258-d263-4df1-aaf0-422eace67cc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:18:12.870090 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.870021    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e419258-d263-4df1-aaf0-422eace67cc9-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e419258-d263-4df1-aaf0-422eace67cc9" (UID: "9e419258-d263-4df1-aaf0-422eace67cc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:18:12.870236 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.870217    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e419258-d263-4df1-aaf0-422eace67cc9-kube-api-access-cvwsm" (OuterVolumeSpecName: "kube-api-access-cvwsm") pod "9e419258-d263-4df1-aaf0-422eace67cc9" (UID: "9e419258-d263-4df1-aaf0-422eace67cc9"). InnerVolumeSpecName "kube-api-access-cvwsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:18:12.870303 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.870287    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e419258-d263-4df1-aaf0-422eace67cc9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e419258-d263-4df1-aaf0-422eace67cc9" (UID: "9e419258-d263-4df1-aaf0-422eace67cc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 11:18:12.969227 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.969189    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e419258-d263-4df1-aaf0-422eace67cc9-istio-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:12.969227 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.969229    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvwsm\" (UniqueName: \"kubernetes.io/projected/9e419258-d263-4df1-aaf0-422eace67cc9-kube-api-access-cvwsm\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:12.969413 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.969247    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e419258-d263-4df1-aaf0-422eace67cc9-istio-envoy\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:12.969413 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.969262    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e419258-d263-4df1-aaf0-422eace67cc9-istio-podinfo\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:12.969413 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:12.969276    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e419258-d263-4df1-aaf0-422eace67cc9-istiod-ca-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:13.290418 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:13.290319    2561 generic.go:356] "Generic (PLEG): container finished" podID="9e419258-d263-4df1-aaf0-422eace67cc9" containerID="02d9283172b7b2ba2a7ab2d8304070e06ad3231d00c0191462787726cca96a55" exitCode=0
Sep 14 11:18:13.290418 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:13.290356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv" event={"ID":"9e419258-d263-4df1-aaf0-422eace67cc9","Type":"ContainerDied","Data":"02d9283172b7b2ba2a7ab2d8304070e06ad3231d00c0191462787726cca96a55"}
Sep 14 11:18:13.290418 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:13.290369    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"
Sep 14 11:18:13.290418 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:13.290411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv" event={"ID":"9e419258-d263-4df1-aaf0-422eace67cc9","Type":"ContainerDied","Data":"7c44e2c4cf7408296d8f7d6104a54617fe28b2a991488d1365a02bfeba903c20"}
Sep 14 11:18:13.290773 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:13.290433    2561 scope.go:117] "RemoveContainer" containerID="02d9283172b7b2ba2a7ab2d8304070e06ad3231d00c0191462787726cca96a55"
Sep 14 11:18:13.300173 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:13.300158    2561 scope.go:117] "RemoveContainer" containerID="71db1e8a767f07cfb39c07d2dd500d7a61a3b6deb07b70eddec204826147ad90"
Sep 14 11:18:13.309100 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:13.309081    2561 scope.go:117] "RemoveContainer" containerID="02d9283172b7b2ba2a7ab2d8304070e06ad3231d00c0191462787726cca96a55"
Sep 14 11:18:13.309362 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:18:13.309342    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d9283172b7b2ba2a7ab2d8304070e06ad3231d00c0191462787726cca96a55\": container with ID starting with 02d9283172b7b2ba2a7ab2d8304070e06ad3231d00c0191462787726cca96a55 not found: ID does not exist" containerID="02d9283172b7b2ba2a7ab2d8304070e06ad3231d00c0191462787726cca96a55"
Sep 14 11:18:13.309407 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:13.309371    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d9283172b7b2ba2a7ab2d8304070e06ad3231d00c0191462787726cca96a55"} err="failed to get container status \"02d9283172b7b2ba2a7ab2d8304070e06ad3231d00c0191462787726cca96a55\": rpc error: code = NotFound desc = could not find container \"02d9283172b7b2ba2a7ab2d8304070e06ad3231d00c0191462787726cca96a55\": container with ID starting with 02d9283172b7b2ba2a7ab2d8304070e06ad3231d00c0191462787726cca96a55 not found: ID does not exist"
Sep 14 11:18:13.309407 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:13.309388    2561 scope.go:117] "RemoveContainer" containerID="71db1e8a767f07cfb39c07d2dd500d7a61a3b6deb07b70eddec204826147ad90"
Sep 14 11:18:13.309640 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:18:13.309623    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71db1e8a767f07cfb39c07d2dd500d7a61a3b6deb07b70eddec204826147ad90\": container with ID starting with 71db1e8a767f07cfb39c07d2dd500d7a61a3b6deb07b70eddec204826147ad90 not found: ID does not exist" containerID="71db1e8a767f07cfb39c07d2dd500d7a61a3b6deb07b70eddec204826147ad90"
Sep 14 11:18:13.309698 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:13.309646    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71db1e8a767f07cfb39c07d2dd500d7a61a3b6deb07b70eddec204826147ad90"} err="failed to get container status \"71db1e8a767f07cfb39c07d2dd500d7a61a3b6deb07b70eddec204826147ad90\": rpc error: code = NotFound desc = could not find container \"71db1e8a767f07cfb39c07d2dd500d7a61a3b6deb07b70eddec204826147ad90\": container with ID starting with 71db1e8a767f07cfb39c07d2dd500d7a61a3b6deb07b70eddec204826147ad90 not found: ID does not exist"
Sep 14 11:18:13.313463 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:13.313443    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"]
Sep 14 11:18:13.317666 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:13.317641    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-22b014-openshift-default-6cbf9d5c54-7h2lv"]
Sep 14 11:18:13.803241 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:13.803209    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e419258-d263-4df1-aaf0-422eace67cc9" path="/var/lib/kubelet/pods/9e419258-d263-4df1-aaf0-422eace67cc9/volumes"
Sep 14 11:18:15.965306 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:15.965276    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 11:18:15.970580 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:15.970556    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 11:18:25.929399 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:25.929359    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:18:35.325604 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:35.325571    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:18:37.736765 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.736734    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7845-d952c8"]
Sep 14 11:18:37.737142 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737114    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f21905f-da93-4a4b-91e4-cabfe6cb9bf0" containerName="maas-api"
Sep 14 11:18:37.737142 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737125    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f21905f-da93-4a4b-91e4-cabfe6cb9bf0" containerName="maas-api"
Sep 14 11:18:37.737142 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737134    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7ed0e43-21e3-40ba-9759-14bd19b53cef" containerName="test-curl-7845-6a81f9"
Sep 14 11:18:37.737142 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737139    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7ed0e43-21e3-40ba-9759-14bd19b53cef" containerName="test-curl-7845-6a81f9"
Sep 14 11:18:37.737279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737152    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67039613-681d-44e6-b719-ba5086dd7823" containerName="maas-api"
Sep 14 11:18:37.737279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737158    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="67039613-681d-44e6-b719-ba5086dd7823" containerName="maas-api"
Sep 14 11:18:37.737279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737166    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerName="istio-proxy"
Sep 14 11:18:37.737279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737172    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerName="istio-proxy"
Sep 14 11:18:37.737279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737181    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e419258-d263-4df1-aaf0-422eace67cc9" containerName="istio-proxy"
Sep 14 11:18:37.737279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737186    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e419258-d263-4df1-aaf0-422eace67cc9" containerName="istio-proxy"
Sep 14 11:18:37.737279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737196    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerName="istio-proxy"
Sep 14 11:18:37.737279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737201    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerName="istio-proxy"
Sep 14 11:18:37.737279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737215    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e419258-d263-4df1-aaf0-422eace67cc9" containerName="istio-proxy"
Sep 14 11:18:37.737279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737220    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e419258-d263-4df1-aaf0-422eace67cc9" containerName="istio-proxy"
Sep 14 11:18:37.737279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737277    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e419258-d263-4df1-aaf0-422eace67cc9" containerName="istio-proxy"
Sep 14 11:18:37.737279 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737284    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerName="istio-proxy"
Sep 14 11:18:37.737639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737291    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="947e8ad6-e7dc-4740-a0e7-7a05962f91f3" containerName="istio-proxy"
Sep 14 11:18:37.737639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737296    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="67039613-681d-44e6-b719-ba5086dd7823" containerName="maas-api"
Sep 14 11:18:37.737639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737302    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f21905f-da93-4a4b-91e4-cabfe6cb9bf0" containerName="maas-api"
Sep 14 11:18:37.737639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.737308    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7ed0e43-21e3-40ba-9759-14bd19b53cef" containerName="test-curl-7845-6a81f9"
Sep 14 11:18:37.740310 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.740295    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7845-d952c8"
Sep 14 11:18:37.743940 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.743921    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-stw9c\""
Sep 14 11:18:37.749478 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.749452    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7845-d952c8"]
Sep 14 11:18:37.893007 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.892977    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfn7n\" (UniqueName: \"kubernetes.io/projected/039051c1-84a1-4c85-b507-0291a00a26b2-kube-api-access-dfn7n\") pod \"test-curl-7845-d952c8\" (UID: \"039051c1-84a1-4c85-b507-0291a00a26b2\") " pod="opendatahub/test-curl-7845-d952c8"
Sep 14 11:18:37.994380 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:37.994311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfn7n\" (UniqueName: \"kubernetes.io/projected/039051c1-84a1-4c85-b507-0291a00a26b2-kube-api-access-dfn7n\") pod \"test-curl-7845-d952c8\" (UID: \"039051c1-84a1-4c85-b507-0291a00a26b2\") " pod="opendatahub/test-curl-7845-d952c8"
Sep 14 11:18:38.003659 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:38.003635    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfn7n\" (UniqueName: \"kubernetes.io/projected/039051c1-84a1-4c85-b507-0291a00a26b2-kube-api-access-dfn7n\") pod \"test-curl-7845-d952c8\" (UID: \"039051c1-84a1-4c85-b507-0291a00a26b2\") " pod="opendatahub/test-curl-7845-d952c8"
Sep 14 11:18:38.050313 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:38.050285    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7845-d952c8"
Sep 14 11:18:38.336661 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:38.336635    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7845-d952c8"]
Sep 14 11:18:38.339348 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:18:38.339316    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod039051c1_84a1_4c85_b507_0291a00a26b2.slice/crio-691f0859662184239af448f2721b553bbccf575605ec73403cd89dc5d83134fd WatchSource:0}: Error finding container 691f0859662184239af448f2721b553bbccf575605ec73403cd89dc5d83134fd: Status 404 returned error can't find the container with id 691f0859662184239af448f2721b553bbccf575605ec73403cd89dc5d83134fd
Sep 14 11:18:38.436025 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:38.435998    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7845-d952c8" event={"ID":"039051c1-84a1-4c85-b507-0291a00a26b2","Type":"ContainerStarted","Data":"7933dd96e1cc25bf8e3624d6beeaa92360e4f7364b5dfa9393e737894c76f1e9"}
Sep 14 11:18:38.436161 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:38.436037    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7845-d952c8" event={"ID":"039051c1-84a1-4c85-b507-0291a00a26b2","Type":"ContainerStarted","Data":"691f0859662184239af448f2721b553bbccf575605ec73403cd89dc5d83134fd"}
Sep 14 11:18:38.461265 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:38.461214    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7845-d952c8" podStartSLOduration=1.461196318 podStartE2EDuration="1.461196318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:18:38.456551373 +0000 UTC m=+1823.141235761" watchObservedRunningTime="2026-09-14 11:18:38.461196318 +0000 UTC m=+1823.145880708"
Sep 14 11:18:39.055402 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.053769    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7845-d952c8"]
Sep 14 11:18:39.057160 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.057131    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7845-d952c8"]
Sep 14 11:18:39.171800 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.171767    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7845-4ebfe6"]
Sep 14 11:18:39.172205 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.172191    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="039051c1-84a1-4c85-b507-0291a00a26b2" containerName="test-curl-7845-d952c8"
Sep 14 11:18:39.172253 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.172206    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="039051c1-84a1-4c85-b507-0291a00a26b2" containerName="test-curl-7845-d952c8"
Sep 14 11:18:39.172290 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.172271    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e419258-d263-4df1-aaf0-422eace67cc9" containerName="istio-proxy"
Sep 14 11:18:39.172290 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.172281    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="039051c1-84a1-4c85-b507-0291a00a26b2" containerName="test-curl-7845-d952c8"
Sep 14 11:18:39.175294 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.175279    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7845-4ebfe6"
Sep 14 11:18:39.182892 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.182834    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7845-4ebfe6"]
Sep 14 11:18:39.203386 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.203355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjk2x\" (UniqueName: \"kubernetes.io/projected/5399a7e5-7f8d-401e-8a30-1320053f7451-kube-api-access-bjk2x\") pod \"test-curl-7845-4ebfe6\" (UID: \"5399a7e5-7f8d-401e-8a30-1320053f7451\") " pod="opendatahub/test-curl-7845-4ebfe6"
Sep 14 11:18:39.304355 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.304322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjk2x\" (UniqueName: \"kubernetes.io/projected/5399a7e5-7f8d-401e-8a30-1320053f7451-kube-api-access-bjk2x\") pod \"test-curl-7845-4ebfe6\" (UID: \"5399a7e5-7f8d-401e-8a30-1320053f7451\") " pod="opendatahub/test-curl-7845-4ebfe6"
Sep 14 11:18:39.314548 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.314492    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjk2x\" (UniqueName: \"kubernetes.io/projected/5399a7e5-7f8d-401e-8a30-1320053f7451-kube-api-access-bjk2x\") pod \"test-curl-7845-4ebfe6\" (UID: \"5399a7e5-7f8d-401e-8a30-1320053f7451\") " pod="opendatahub/test-curl-7845-4ebfe6"
Sep 14 11:18:39.441539 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:18:39.441514    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7845-d952c8" container="test-curl-7845-d952c8"
Sep 14 11:18:39.486371 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.486347    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7845-4ebfe6"
Sep 14 11:18:39.761896 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:39.761869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7845-4ebfe6"]
Sep 14 11:18:39.762747 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:18:39.762702    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5399a7e5_7f8d_401e_8a30_1320053f7451.slice/crio-6dda94a2f59c9371a43027bb92600d5c7b099c5cf8c4f4c10986bc88fb3c180c WatchSource:0}: Error finding container 6dda94a2f59c9371a43027bb92600d5c7b099c5cf8c4f4c10986bc88fb3c180c: Status 404 returned error can't find the container with id 6dda94a2f59c9371a43027bb92600d5c7b099c5cf8c4f4c10986bc88fb3c180c
Sep 14 11:18:40.448340 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:40.448302    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7845-4ebfe6" event={"ID":"5399a7e5-7f8d-401e-8a30-1320053f7451","Type":"ContainerStarted","Data":"2c1a11da4da7712cf0f69ad461bede2a7b17d859188e6977e8d9162a292d734c"}
Sep 14 11:18:40.448340 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:40.448344    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7845-4ebfe6" event={"ID":"5399a7e5-7f8d-401e-8a30-1320053f7451","Type":"ContainerStarted","Data":"6dda94a2f59c9371a43027bb92600d5c7b099c5cf8c4f4c10986bc88fb3c180c"}
Sep 14 11:18:40.448943 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:40.448367    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7845-d952c8" podUID="039051c1-84a1-4c85-b507-0291a00a26b2" containerName="test-curl-7845-d952c8" containerID="cri-o://7933dd96e1cc25bf8e3624d6beeaa92360e4f7364b5dfa9393e737894c76f1e9" gracePeriod=2
Sep 14 11:18:40.468417 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:40.468369    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7845-4ebfe6" podStartSLOduration=1.4683550730000001 podStartE2EDuration="1.468355073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:18:40.466566819 +0000 UTC m=+1825.151251209" watchObservedRunningTime="2026-09-14 11:18:40.468355073 +0000 UTC m=+1825.153039461"
Sep 14 11:18:41.072221 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:41.072191    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7845-4ebfe6"]
Sep 14 11:18:41.078692 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:41.078663    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7845-4ebfe6"]
Sep 14 11:18:41.454792 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:18:41.454763    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7845-4ebfe6" container="test-curl-7845-4ebfe6"
Sep 14 11:18:42.463525 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:42.463486    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7845-4ebfe6" podUID="5399a7e5-7f8d-401e-8a30-1320053f7451" containerName="test-curl-7845-4ebfe6" containerID="cri-o://2c1a11da4da7712cf0f69ad461bede2a7b17d859188e6977e8d9162a292d734c" gracePeriod=2
Sep 14 11:18:42.622267 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:42.622245    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7845-d952c8"
Sep 14 11:18:42.634529 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:42.634509    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfn7n\" (UniqueName: \"kubernetes.io/projected/039051c1-84a1-4c85-b507-0291a00a26b2-kube-api-access-dfn7n\") pod \"039051c1-84a1-4c85-b507-0291a00a26b2\" (UID: \"039051c1-84a1-4c85-b507-0291a00a26b2\") "
Sep 14 11:18:42.636396 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:42.636374    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039051c1-84a1-4c85-b507-0291a00a26b2-kube-api-access-dfn7n" (OuterVolumeSpecName: "kube-api-access-dfn7n") pod "039051c1-84a1-4c85-b507-0291a00a26b2" (UID: "039051c1-84a1-4c85-b507-0291a00a26b2"). InnerVolumeSpecName "kube-api-access-dfn7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:18:42.736207 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:42.736150    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfn7n\" (UniqueName: \"kubernetes.io/projected/039051c1-84a1-4c85-b507-0291a00a26b2-kube-api-access-dfn7n\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:43.469866 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:43.469828    2561 generic.go:356] "Generic (PLEG): container finished" podID="039051c1-84a1-4c85-b507-0291a00a26b2" containerID="7933dd96e1cc25bf8e3624d6beeaa92360e4f7364b5dfa9393e737894c76f1e9" exitCode=137
Sep 14 11:18:43.469866 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:43.469867    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7845-d952c8"
Sep 14 11:18:43.470332 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:43.469933    2561 scope.go:117] "RemoveContainer" containerID="7933dd96e1cc25bf8e3624d6beeaa92360e4f7364b5dfa9393e737894c76f1e9"
Sep 14 11:18:43.479438 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:43.479421    2561 scope.go:117] "RemoveContainer" containerID="7933dd96e1cc25bf8e3624d6beeaa92360e4f7364b5dfa9393e737894c76f1e9"
Sep 14 11:18:43.479703 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:18:43.479686    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7933dd96e1cc25bf8e3624d6beeaa92360e4f7364b5dfa9393e737894c76f1e9\": container with ID starting with 7933dd96e1cc25bf8e3624d6beeaa92360e4f7364b5dfa9393e737894c76f1e9 not found: ID does not exist" containerID="7933dd96e1cc25bf8e3624d6beeaa92360e4f7364b5dfa9393e737894c76f1e9"
Sep 14 11:18:43.479801 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:43.479739    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7933dd96e1cc25bf8e3624d6beeaa92360e4f7364b5dfa9393e737894c76f1e9"} err="failed to get container status \"7933dd96e1cc25bf8e3624d6beeaa92360e4f7364b5dfa9393e737894c76f1e9\": rpc error: code = NotFound desc = could not find container \"7933dd96e1cc25bf8e3624d6beeaa92360e4f7364b5dfa9393e737894c76f1e9\": container with ID starting with 7933dd96e1cc25bf8e3624d6beeaa92360e4f7364b5dfa9393e737894c76f1e9 not found: ID does not exist"
Sep 14 11:18:43.481836 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:18:43.481811    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7845-d952c8\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="039051c1-84a1-4c85-b507-0291a00a26b2" pod="opendatahub/test-curl-7845-d952c8"
Sep 14 11:18:43.803811 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:43.803734    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="039051c1-84a1-4c85-b507-0291a00a26b2" path="/var/lib/kubelet/pods/039051c1-84a1-4c85-b507-0291a00a26b2/volumes"
Sep 14 11:18:44.601636 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:44.601614    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7845-4ebfe6"
Sep 14 11:18:44.652296 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:44.652274    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjk2x\" (UniqueName: \"kubernetes.io/projected/5399a7e5-7f8d-401e-8a30-1320053f7451-kube-api-access-bjk2x\") pod \"5399a7e5-7f8d-401e-8a30-1320053f7451\" (UID: \"5399a7e5-7f8d-401e-8a30-1320053f7451\") "
Sep 14 11:18:44.654181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:44.654149    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5399a7e5-7f8d-401e-8a30-1320053f7451-kube-api-access-bjk2x" (OuterVolumeSpecName: "kube-api-access-bjk2x") pod "5399a7e5-7f8d-401e-8a30-1320053f7451" (UID: "5399a7e5-7f8d-401e-8a30-1320053f7451"). InnerVolumeSpecName "kube-api-access-bjk2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:18:44.753266 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:44.753198    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjk2x\" (UniqueName: \"kubernetes.io/projected/5399a7e5-7f8d-401e-8a30-1320053f7451-kube-api-access-bjk2x\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:18:45.484462 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:45.484423    2561 generic.go:356] "Generic (PLEG): container finished" podID="5399a7e5-7f8d-401e-8a30-1320053f7451" containerID="2c1a11da4da7712cf0f69ad461bede2a7b17d859188e6977e8d9162a292d734c" exitCode=137
Sep 14 11:18:45.484642 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:45.484477    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7845-4ebfe6"
Sep 14 11:18:45.484642 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:45.484526    2561 scope.go:117] "RemoveContainer" containerID="2c1a11da4da7712cf0f69ad461bede2a7b17d859188e6977e8d9162a292d734c"
Sep 14 11:18:45.494955 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:45.494933    2561 scope.go:117] "RemoveContainer" containerID="2c1a11da4da7712cf0f69ad461bede2a7b17d859188e6977e8d9162a292d734c"
Sep 14 11:18:45.495244 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:18:45.495221    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c1a11da4da7712cf0f69ad461bede2a7b17d859188e6977e8d9162a292d734c\": container with ID starting with 2c1a11da4da7712cf0f69ad461bede2a7b17d859188e6977e8d9162a292d734c not found: ID does not exist" containerID="2c1a11da4da7712cf0f69ad461bede2a7b17d859188e6977e8d9162a292d734c"
Sep 14 11:18:45.495302 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:45.495252    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c1a11da4da7712cf0f69ad461bede2a7b17d859188e6977e8d9162a292d734c"} err="failed to get container status \"2c1a11da4da7712cf0f69ad461bede2a7b17d859188e6977e8d9162a292d734c\": rpc error: code = NotFound desc = could not find container \"2c1a11da4da7712cf0f69ad461bede2a7b17d859188e6977e8d9162a292d734c\": container with ID starting with 2c1a11da4da7712cf0f69ad461bede2a7b17d859188e6977e8d9162a292d734c not found: ID does not exist"
Sep 14 11:18:45.495550 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:18:45.495528    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7845-4ebfe6\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="5399a7e5-7f8d-401e-8a30-1320053f7451" pod="opendatahub/test-curl-7845-4ebfe6"
Sep 14 11:18:45.802135 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:18:45.802048    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7845-4ebfe6\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="5399a7e5-7f8d-401e-8a30-1320053f7451" pod="opendatahub/test-curl-7845-4ebfe6"
Sep 14 11:18:45.803016 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:45.802996    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5399a7e5-7f8d-401e-8a30-1320053f7451" path="/var/lib/kubelet/pods/5399a7e5-7f8d-401e-8a30-1320053f7451/volumes"
Sep 14 11:18:46.123591 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:46.123562    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:18:50.755335 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.755297    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"]
Sep 14 11:18:50.755772 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.755757    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5399a7e5-7f8d-401e-8a30-1320053f7451" containerName="test-curl-7845-4ebfe6"
Sep 14 11:18:50.755837 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.755774    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5399a7e5-7f8d-401e-8a30-1320053f7451" containerName="test-curl-7845-4ebfe6"
Sep 14 11:18:50.755882 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.755839    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5399a7e5-7f8d-401e-8a30-1320053f7451" containerName="test-curl-7845-4ebfe6"
Sep 14 11:18:50.758951 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.758935    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.762565 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.762543    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fdbc00\"/\"default-dockercfg-xwbwm\""
Sep 14 11:18:50.763674 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.763655    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fdbc00\"/\"kube-root-ca.crt\""
Sep 14 11:18:50.764951 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.764931    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fdbc00\"/\"openshift-service-ca.crt\""
Sep 14 11:18:50.765080 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.765062    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fdbc00\"/\"rate-test-model-fdbc00-kserve-self-signed-certs\""
Sep 14 11:18:50.773391 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.773370    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"]
Sep 14 11:18:50.809000 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.808968    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a905e1-10ed-4c6d-926c-a58640e3d175-tls-certs\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.809102 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.809007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-model-cache\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.809102 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.809048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55p9\" (UniqueName: \"kubernetes.io/projected/e7a905e1-10ed-4c6d-926c-a58640e3d175-kube-api-access-l55p9\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.809200 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.809129    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-dshm\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.809200 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.809181    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-tmp-dir\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.809272 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.809253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-home\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.910389 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.910356    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a905e1-10ed-4c6d-926c-a58640e3d175-tls-certs\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.910389 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.910393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-model-cache\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.910631 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.910419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55p9\" (UniqueName: \"kubernetes.io/projected/e7a905e1-10ed-4c6d-926c-a58640e3d175-kube-api-access-l55p9\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.910631 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.910451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-dshm\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.910631 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.910483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-tmp-dir\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.910631 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.910530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-home\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.910895 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.910862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-model-cache\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.910980 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.910917    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-tmp-dir\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.911107 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.911089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-home\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.912802 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.912780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-dshm\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.912964 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.912948    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a905e1-10ed-4c6d-926c-a58640e3d175-tls-certs\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:50.921889 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:50.921870    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55p9\" (UniqueName: \"kubernetes.io/projected/e7a905e1-10ed-4c6d-926c-a58640e3d175-kube-api-access-l55p9\") pod \"rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:51.067787 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:51.067689    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:18:51.342383 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:51.342357    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"]
Sep 14 11:18:51.343802 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:18:51.343764    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7a905e1_10ed_4c6d_926c_a58640e3d175.slice/crio-7d067e967265c032e59c2d68956fa61d36f6afa4dbd9186e29c45080c33563bc WatchSource:0}: Error finding container 7d067e967265c032e59c2d68956fa61d36f6afa4dbd9186e29c45080c33563bc: Status 404 returned error can't find the container with id 7d067e967265c032e59c2d68956fa61d36f6afa4dbd9186e29c45080c33563bc
Sep 14 11:18:51.521500 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:51.521467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg" event={"ID":"e7a905e1-10ed-4c6d-926c-a58640e3d175","Type":"ContainerStarted","Data":"fc568b158efcdae76c3e01ed917749b22aee64a0a1c04b2d81c7579153a44658"}
Sep 14 11:18:51.521500 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:51.521507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg" event={"ID":"e7a905e1-10ed-4c6d-926c-a58640e3d175","Type":"ContainerStarted","Data":"7d067e967265c032e59c2d68956fa61d36f6afa4dbd9186e29c45080c33563bc"}
Sep 14 11:18:51.550764 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:18:51.550706    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg" podStartSLOduration=1.550691268 podStartE2EDuration="1.550691268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:18:51.546054839 +0000 UTC m=+1836.230739230" watchObservedRunningTime="2026-09-14 11:18:51.550691268 +0000 UTC m=+1836.235375650"
Sep 14 11:19:00.640963 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:00.640933    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:19:01.068203 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:01.068111    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:19:01.068398 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:01.068375    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:19:01.080652 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:01.080630    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:19:01.588922 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:01.588888    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:19:16.218486 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:16.218460    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:19:36.343056 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:36.343021    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:19:45.730429 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:45.730341    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:19:52.585224 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:52.585190    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"]
Sep 14 11:19:56.044288 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:56.044251    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:19:59.940366 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:59.940328    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"]
Sep 14 11:19:59.944170 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:59.944152    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:19:59.947458 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:59.947436    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bf45e0-serving-cert\""
Sep 14 11:19:59.964592 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:19:59.964565    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"]
Sep 14 11:20:00.042865 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.042825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca88e641-e8f3-4f23-9357-fc034752428b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:00.042865 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.042877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca88e641-e8f3-4f23-9357-fc034752428b-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:00.043076 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.042975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nhlk\" (UniqueName: \"kubernetes.io/projected/ca88e641-e8f3-4f23-9357-fc034752428b-kube-api-access-4nhlk\") pod \"maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:00.043076 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.043044    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca88e641-e8f3-4f23-9357-fc034752428b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:00.144523 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.144494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca88e641-e8f3-4f23-9357-fc034752428b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:00.144756 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.144547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca88e641-e8f3-4f23-9357-fc034752428b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:00.144756 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.144639    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca88e641-e8f3-4f23-9357-fc034752428b-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:00.144756 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.144677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nhlk\" (UniqueName: \"kubernetes.io/projected/ca88e641-e8f3-4f23-9357-fc034752428b-kube-api-access-4nhlk\") pod \"maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:00.145396 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.145372    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca88e641-e8f3-4f23-9357-fc034752428b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:00.147085 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.147056    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca88e641-e8f3-4f23-9357-fc034752428b-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:00.147209 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.147102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca88e641-e8f3-4f23-9357-fc034752428b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:00.156389 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.156363    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nhlk\" (UniqueName: \"kubernetes.io/projected/ca88e641-e8f3-4f23-9357-fc034752428b-kube-api-access-4nhlk\") pod \"maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:00.254770 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.254656    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:00.544304 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:20:00.544260    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca88e641_e8f3_4f23_9357_fc034752428b.slice/crio-d5bec1bbe1ec3abc424002149bae08cd3ae98a1e76ceca4e2e01519d63753d94 WatchSource:0}: Error finding container d5bec1bbe1ec3abc424002149bae08cd3ae98a1e76ceca4e2e01519d63753d94: Status 404 returned error can't find the container with id d5bec1bbe1ec3abc424002149bae08cd3ae98a1e76ceca4e2e01519d63753d94
Sep 14 11:20:00.545679 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.545654    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"]
Sep 14 11:20:00.918163 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:00.918132    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4" event={"ID":"ca88e641-e8f3-4f23-9357-fc034752428b","Type":"ContainerStarted","Data":"d5bec1bbe1ec3abc424002149bae08cd3ae98a1e76ceca4e2e01519d63753d94"}
Sep 14 11:20:01.925491 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:01.925450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4" event={"ID":"ca88e641-e8f3-4f23-9357-fc034752428b","Type":"ContainerStarted","Data":"12e041fdd6fc1f792d0ab5f9b55a9bbac24c7a202857430aa87177135ece8b62"}
Sep 14 11:20:01.925982 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:01.925520    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:07.604832 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:07.604788    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg" podUID="e7a905e1-10ed-4c6d-926c-a58640e3d175" containerName="main" containerID="cri-o://fc568b158efcdae76c3e01ed917749b22aee64a0a1c04b2d81c7579153a44658" gracePeriod=45
Sep 14 11:20:07.859072 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:07.859014    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:20:07.882990 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:07.882935    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4" podStartSLOduration=8.285933283 podStartE2EDuration="8.882914123s" totalImagesPullingTime="596.98084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:20:01.987916268 +0000 UTC m=+1906.672600656" watchObservedRunningTime="2026-09-14 11:20:07.882914123 +0000 UTC m=+1912.567598524"
Sep 14 11:20:07.935794 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:07.935770    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:20:07.960741 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:07.960698    2561 generic.go:356] "Generic (PLEG): container finished" podID="e7a905e1-10ed-4c6d-926c-a58640e3d175" containerID="fc568b158efcdae76c3e01ed917749b22aee64a0a1c04b2d81c7579153a44658" exitCode=0
Sep 14 11:20:07.960859 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:07.960762    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"
Sep 14 11:20:07.960859 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:07.960790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg" event={"ID":"e7a905e1-10ed-4c6d-926c-a58640e3d175","Type":"ContainerDied","Data":"fc568b158efcdae76c3e01ed917749b22aee64a0a1c04b2d81c7579153a44658"}
Sep 14 11:20:07.960859 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:07.960831    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg" event={"ID":"e7a905e1-10ed-4c6d-926c-a58640e3d175","Type":"ContainerDied","Data":"7d067e967265c032e59c2d68956fa61d36f6afa4dbd9186e29c45080c33563bc"}
Sep 14 11:20:07.960859 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:07.960848    2561 scope.go:117] "RemoveContainer" containerID="fc568b158efcdae76c3e01ed917749b22aee64a0a1c04b2d81c7579153a44658"
Sep 14 11:20:07.970385 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:07.970342    2561 scope.go:117] "RemoveContainer" containerID="fc568b158efcdae76c3e01ed917749b22aee64a0a1c04b2d81c7579153a44658"
Sep 14 11:20:07.970681 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:20:07.970653    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc568b158efcdae76c3e01ed917749b22aee64a0a1c04b2d81c7579153a44658\": container with ID starting with fc568b158efcdae76c3e01ed917749b22aee64a0a1c04b2d81c7579153a44658 not found: ID does not exist" containerID="fc568b158efcdae76c3e01ed917749b22aee64a0a1c04b2d81c7579153a44658"
Sep 14 11:20:07.970824 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:07.970689    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc568b158efcdae76c3e01ed917749b22aee64a0a1c04b2d81c7579153a44658"} err="failed to get container status \"fc568b158efcdae76c3e01ed917749b22aee64a0a1c04b2d81c7579153a44658\": rpc error: code = NotFound desc = could not find container \"fc568b158efcdae76c3e01ed917749b22aee64a0a1c04b2d81c7579153a44658\": container with ID starting with fc568b158efcdae76c3e01ed917749b22aee64a0a1c04b2d81c7579153a44658 not found: ID does not exist"
Sep 14 11:20:08.018238 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.018211    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-dshm\") pod \"e7a905e1-10ed-4c6d-926c-a58640e3d175\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") "
Sep 14 11:20:08.018393 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.018247    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-tmp-dir\") pod \"e7a905e1-10ed-4c6d-926c-a58640e3d175\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") "
Sep 14 11:20:08.018393 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.018276    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a905e1-10ed-4c6d-926c-a58640e3d175-tls-certs\") pod \"e7a905e1-10ed-4c6d-926c-a58640e3d175\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") "
Sep 14 11:20:08.018393 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.018336    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-model-cache\") pod \"e7a905e1-10ed-4c6d-926c-a58640e3d175\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") "
Sep 14 11:20:08.018583 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.018436    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-home\") pod \"e7a905e1-10ed-4c6d-926c-a58640e3d175\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") "
Sep 14 11:20:08.018583 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.018490    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l55p9\" (UniqueName: \"kubernetes.io/projected/e7a905e1-10ed-4c6d-926c-a58640e3d175-kube-api-access-l55p9\") pod \"e7a905e1-10ed-4c6d-926c-a58640e3d175\" (UID: \"e7a905e1-10ed-4c6d-926c-a58640e3d175\") "
Sep 14 11:20:08.018583 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.018522    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7a905e1-10ed-4c6d-926c-a58640e3d175" (UID: "e7a905e1-10ed-4c6d-926c-a58640e3d175"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:20:08.018754 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.018681    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7a905e1-10ed-4c6d-926c-a58640e3d175" (UID: "e7a905e1-10ed-4c6d-926c-a58640e3d175"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:20:08.018819 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.018755    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-home" (OuterVolumeSpecName: "home") pod "e7a905e1-10ed-4c6d-926c-a58640e3d175" (UID: "e7a905e1-10ed-4c6d-926c-a58640e3d175"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:20:08.019183 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.019143    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-home\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:20:08.019295 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.019186    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-tmp-dir\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:20:08.019295 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.019202    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-model-cache\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:20:08.020572 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.020545    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-dshm" (OuterVolumeSpecName: "dshm") pod "e7a905e1-10ed-4c6d-926c-a58640e3d175" (UID: "e7a905e1-10ed-4c6d-926c-a58640e3d175"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:20:08.020697 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.020631    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7a905e1-10ed-4c6d-926c-a58640e3d175-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7a905e1-10ed-4c6d-926c-a58640e3d175" (UID: "e7a905e1-10ed-4c6d-926c-a58640e3d175"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:20:08.020819 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.020749    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a905e1-10ed-4c6d-926c-a58640e3d175-kube-api-access-l55p9" (OuterVolumeSpecName: "kube-api-access-l55p9") pod "e7a905e1-10ed-4c6d-926c-a58640e3d175" (UID: "e7a905e1-10ed-4c6d-926c-a58640e3d175"). InnerVolumeSpecName "kube-api-access-l55p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:20:08.120526 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.120430    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l55p9\" (UniqueName: \"kubernetes.io/projected/e7a905e1-10ed-4c6d-926c-a58640e3d175-kube-api-access-l55p9\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:20:08.120526 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.120460    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a905e1-10ed-4c6d-926c-a58640e3d175-dshm\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:20:08.120526 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.120472    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a905e1-10ed-4c6d-926c-a58640e3d175-tls-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:20:08.290627 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.290596    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"]
Sep 14 11:20:08.295483 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:08.295460    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/rate-test-model-fdbc00-kserve-644d9f4f55-ljqgg"]
Sep 14 11:20:09.204519 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:09.204486    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:20:09.803679 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:09.803645    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7a905e1-10ed-4c6d-926c-a58640e3d175" path="/var/lib/kubelet/pods/e7a905e1-10ed-4c6d-926c-a58640e3d175/volumes"
Sep 14 11:20:23.590379 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.590345    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"]
Sep 14 11:20:23.590980 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.590783    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a905e1-10ed-4c6d-926c-a58640e3d175" containerName="main"
Sep 14 11:20:23.590980 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.590795    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a905e1-10ed-4c6d-926c-a58640e3d175" containerName="main"
Sep 14 11:20:23.590980 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.590863    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7a905e1-10ed-4c6d-926c-a58640e3d175" containerName="main"
Sep 14 11:20:23.593925 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.593908    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.596746 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.596722    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a521b5-openshift-default-dockercfg-nmtqc\""
Sep 14 11:20:23.611676 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.611652    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"]
Sep 14 11:20:23.661626 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.661603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-istio-data\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.661773 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.661638    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-credential-socket\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.661773 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.661663    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196e2ccf-eb11-4841-b102-7e600293c333-istio-podinfo\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.661773 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.661685    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196e2ccf-eb11-4841-b102-7e600293c333-istiod-ca-cert\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.661773 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.661762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-workload-certs\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.661973 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.661780    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196e2ccf-eb11-4841-b102-7e600293c333-istio-token\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.661973 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.661799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-workload-socket\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.661973 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.661817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v99gf\" (UniqueName: \"kubernetes.io/projected/196e2ccf-eb11-4841-b102-7e600293c333-kube-api-access-v99gf\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.661973 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.661932    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-istio-envoy\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.762767 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.762698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-istio-envoy\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.762958 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.762796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-istio-data\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.762958 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.762830    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-credential-socket\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.762958 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.762863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196e2ccf-eb11-4841-b102-7e600293c333-istio-podinfo\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.762958 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.762899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196e2ccf-eb11-4841-b102-7e600293c333-istiod-ca-cert\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.763193 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.762960    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-workload-certs\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.763193 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.762985    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196e2ccf-eb11-4841-b102-7e600293c333-istio-token\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.763193 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.763015    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-workload-socket\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.763193 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.763043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v99gf\" (UniqueName: \"kubernetes.io/projected/196e2ccf-eb11-4841-b102-7e600293c333-kube-api-access-v99gf\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.763398 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.763246    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-istio-data\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.763398 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.763285    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-credential-socket\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.763512 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.763453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-workload-certs\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.763570 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.763546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-workload-socket\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.763843 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.763816    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196e2ccf-eb11-4841-b102-7e600293c333-istiod-ca-cert\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.765135 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.765109    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-istio-envoy\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.765356 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.765340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196e2ccf-eb11-4841-b102-7e600293c333-istio-podinfo\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.771631 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.771610    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196e2ccf-eb11-4841-b102-7e600293c333-istio-token\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.771758 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.771643    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v99gf\" (UniqueName: \"kubernetes.io/projected/196e2ccf-eb11-4841-b102-7e600293c333-kube-api-access-v99gf\") pod \"e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") " pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:23.907390 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:23.907359    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:24.185133 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:24.185106    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"]
Sep 14 11:20:24.186953 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:20:24.186927    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196e2ccf_eb11_4841_b102_7e600293c333.slice/crio-0f332624539fd34865bf98d0fb71653792cba2cc6f99c1f07ab891c759c31c3b WatchSource:0}: Error finding container 0f332624539fd34865bf98d0fb71653792cba2cc6f99c1f07ab891c759c31c3b: Status 404 returned error can't find the container with id 0f332624539fd34865bf98d0fb71653792cba2cc6f99c1f07ab891c759c31c3b
Sep 14 11:20:24.189467 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:24.189434    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:20:24.189575 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:24.189515    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:20:24.189575 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:24.189560    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:20:25.063505 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:25.063472    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" event={"ID":"196e2ccf-eb11-4841-b102-7e600293c333","Type":"ContainerStarted","Data":"650900a9f666adea08b7f2fdc685646b60c3526e4963724b25ee9c0e81a0dd1d"}
Sep 14 11:20:25.063909 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:25.063526    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" event={"ID":"196e2ccf-eb11-4841-b102-7e600293c333","Type":"ContainerStarted","Data":"0f332624539fd34865bf98d0fb71653792cba2cc6f99c1f07ab891c759c31c3b"}
Sep 14 11:20:25.088287 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:25.088230    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" podStartSLOduration=2.088213105 podStartE2EDuration="2.088213105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:20:25.085102283 +0000 UTC m=+1929.769786684" watchObservedRunningTime="2026-09-14 11:20:25.088213105 +0000 UTC m=+1929.772897494"
Sep 14 11:20:25.908060 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:25.908029    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:25.913026 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:25.912996    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:26.069805 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:26.069770    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:26.070661 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:26.070641    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:30.009183 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.009149    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"]
Sep 14 11:20:30.012965 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.012943    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.020846 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.020818    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a521b5-serving-cert\""
Sep 14 11:20:30.042976 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.042935    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"]
Sep 14 11:20:30.124555 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.124519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.124756 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.124644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.124756 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.124722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-metrics-certs\") pod \"maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.124756 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.124742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwpqb\" (UniqueName: \"kubernetes.io/projected/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-kube-api-access-cwpqb\") pod \"maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.225669 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.225631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.225853 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.225736    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.225853 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.225813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-metrics-certs\") pod \"maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.225853 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.225838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwpqb\" (UniqueName: \"kubernetes.io/projected/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-kube-api-access-cwpqb\") pod \"maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.225982 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:20:30.225892    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-serving-cert: secret "maas-api-e2e-ipp-b-a521b5-serving-cert" not found
Sep 14 11:20:30.225982 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:20:30.225967    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-maas-api-tls podName:1b99baa9-b81b-446e-b4f3-e5fad2d6e693 nodeName:}" failed. No retries permitted until 2026-09-14 11:20:30.725950882 +0000 UTC m=+1935.410635249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-maas-api-tls") pod "maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8" (UID: "1b99baa9-b81b-446e-b4f3-e5fad2d6e693") : secret "maas-api-e2e-ipp-b-a521b5-serving-cert" not found
Sep 14 11:20:30.226391 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.226373    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.228234 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.228213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-metrics-certs\") pod \"maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.240703 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.240676    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwpqb\" (UniqueName: \"kubernetes.io/projected/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-kube-api-access-cwpqb\") pod \"maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.730324 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.730287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.732677 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.732646    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:30.923790 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:30.923753    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:31.196577 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:31.196553    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"]
Sep 14 11:20:31.198503 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:20:31.198474    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b99baa9_b81b_446e_b4f3_e5fad2d6e693.slice/crio-1c003970f69828cca2924184bcfd9791c0107246420672b0a057d26d89f0085c WatchSource:0}: Error finding container 1c003970f69828cca2924184bcfd9791c0107246420672b0a057d26d89f0085c: Status 404 returned error can't find the container with id 1c003970f69828cca2924184bcfd9791c0107246420672b0a057d26d89f0085c
Sep 14 11:20:31.199811 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:31.199795    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 11:20:32.106032 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:32.105996    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8" event={"ID":"1b99baa9-b81b-446e-b4f3-e5fad2d6e693","Type":"ContainerStarted","Data":"c49ab3538e0514d9c8e2038795ca177f8f38ef38ba2a14267d997b05bd2d938b"}
Sep 14 11:20:32.106191 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:32.106064    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:32.106191 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:32.106087    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8" event={"ID":"1b99baa9-b81b-446e-b4f3-e5fad2d6e693","Type":"ContainerStarted","Data":"1c003970f69828cca2924184bcfd9791c0107246420672b0a057d26d89f0085c"}
Sep 14 11:20:32.124458 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:32.124408    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8" podStartSLOduration=2.569654642 podStartE2EDuration="3.124388938s" totalImagesPullingTime="554.734296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:20:32.124055189 +0000 UTC m=+1936.808739580" watchObservedRunningTime="2026-09-14 11:20:32.124388938 +0000 UTC m=+1936.809073343"
Sep 14 11:20:34.215929 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:34.215890    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:20:38.116882 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:38.116845    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:20:45.193867 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:45.193832    2561 generic.go:356] "Generic (PLEG): container finished" podID="196e2ccf-eb11-4841-b102-7e600293c333" containerID="650900a9f666adea08b7f2fdc685646b60c3526e4963724b25ee9c0e81a0dd1d" exitCode=0
Sep 14 11:20:45.194326 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:45.193888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" event={"ID":"196e2ccf-eb11-4841-b102-7e600293c333","Type":"ContainerDied","Data":"650900a9f666adea08b7f2fdc685646b60c3526e4963724b25ee9c0e81a0dd1d"}
Sep 14 11:20:45.194326 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:45.194250    2561 scope.go:117] "RemoveContainer" containerID="650900a9f666adea08b7f2fdc685646b60c3526e4963724b25ee9c0e81a0dd1d"
Sep 14 11:20:45.195464 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:45.195433    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:20:45.195574 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:45.195504    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:20:45.195574 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:45.195535    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 11:20:46.201949 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:46.201915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" event={"ID":"196e2ccf-eb11-4841-b102-7e600293c333","Type":"ContainerStarted","Data":"37b79dfc52a24a1d9451d36cf594ea2766081f398a3cbba6123e03997a50c6da"}
Sep 14 11:20:46.908418 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:46.908389    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:46.908622 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:46.908601    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 14 11:20:46.908672 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:46.908652    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" podUID="196e2ccf-eb11-4841-b102-7e600293c333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 14 11:20:47.908019 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:47.907974    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 14 11:20:47.908419 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:47.908036    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" podUID="196e2ccf-eb11-4841-b102-7e600293c333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 14 11:20:48.908256 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:48.908213    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 14 11:20:48.908655 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:48.908296    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" podUID="196e2ccf-eb11-4841-b102-7e600293c333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 14 11:20:49.908581 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:49.908541    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 14 11:20:49.909001 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:49.908629    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" podUID="196e2ccf-eb11-4841-b102-7e600293c333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 14 11:20:50.908243 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:50.908206    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 14 11:20:50.908407 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:50.908262    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" podUID="196e2ccf-eb11-4841-b102-7e600293c333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 14 11:20:51.908133 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:51.908096    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 14 11:20:51.908519 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:51.908165    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" podUID="196e2ccf-eb11-4841-b102-7e600293c333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 14 11:20:52.912707 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:52.912675    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:52.913082 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:52.913063    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:20:52.913654 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:20:52.913637    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:21:15.113568 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:15.113487    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:21:16.206361 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:16.206329    2561 scope.go:117] "RemoveContainer" containerID="7a59b6c9a48aaca3ae20ee0212c6161b3105163999db153937330bf76bb7b925"
Sep 14 11:21:36.916240 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:36.916208    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:21:51.979731 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:51.979689    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"]
Sep 14 11:21:51.983443 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:51.983422    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:51.986305 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:51.986282    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-d5e418-serving-cert\""
Sep 14 11:21:51.993612 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:51.993589    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"]
Sep 14 11:21:52.070348 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.070323    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:52.070506 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.070391    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:52.070506 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.070419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6f66\" (UniqueName: \"kubernetes.io/projected/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-kube-api-access-c6f66\") pod \"maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:52.070506 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.070450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:52.171506 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.171475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:52.171698 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.171625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:52.171698 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.171679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:52.171698 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.171695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6f66\" (UniqueName: \"kubernetes.io/projected/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-kube-api-access-c6f66\") pod \"maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:52.172302 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.172281    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:52.174077 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.174057    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:52.174227 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.174208    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:52.180808 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.180786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6f66\" (UniqueName: \"kubernetes.io/projected/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-kube-api-access-c6f66\") pod \"maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:52.295042 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.294971    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:52.579317 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.579284    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"]
Sep 14 11:21:52.580648 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:21:52.580613    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f5c0bb_ed20_4eaa_a08c_621b2d2a9151.slice/crio-b7c12c24261aae188ea3bebe5b1b517932aa2c4e4aa186c95bed7c56d58ce624 WatchSource:0}: Error finding container b7c12c24261aae188ea3bebe5b1b517932aa2c4e4aa186c95bed7c56d58ce624: Status 404 returned error can't find the container with id b7c12c24261aae188ea3bebe5b1b517932aa2c4e4aa186c95bed7c56d58ce624
Sep 14 11:21:52.592909 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:52.592875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr" event={"ID":"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151","Type":"ContainerStarted","Data":"b7c12c24261aae188ea3bebe5b1b517932aa2c4e4aa186c95bed7c56d58ce624"}
Sep 14 11:21:53.600304 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:53.600265    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr" event={"ID":"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151","Type":"ContainerStarted","Data":"bbf18ef2f38729a3d47e04c2ab5399f05ac614e1176442de0643c2e772ee148a"}
Sep 14 11:21:53.600683 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:53.600363    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:21:53.621024 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:53.620979    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr" podStartSLOduration=2.24389492 podStartE2EDuration="2.62096539s" totalImagesPullingTime="377.07047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:21:53.617811814 +0000 UTC m=+2018.302496204" watchObservedRunningTime="2026-09-14 11:21:53.62096539 +0000 UTC m=+2018.305649779"
Sep 14 11:21:56.411267 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:56.411232    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:21:59.610888 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:21:59.610860    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:22:29.568325 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.568292    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"]
Sep 14 11:22:29.568908 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.568526    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr" podUID="11f5c0bb-ed20-4eaa-a08c-621b2d2a9151" containerName="maas-api" containerID="cri-o://bbf18ef2f38729a3d47e04c2ab5399f05ac614e1176442de0643c2e772ee148a" gracePeriod=30
Sep 14 11:22:29.606183 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.606147    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr" podUID="11f5c0bb-ed20-4eaa-a08c-621b2d2a9151" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.94:8443/health\": dial tcp 10.132.0.94:8443: connect: connection refused"
Sep 14 11:22:29.812130 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.812099    2561 generic.go:356] "Generic (PLEG): container finished" podID="11f5c0bb-ed20-4eaa-a08c-621b2d2a9151" containerID="bbf18ef2f38729a3d47e04c2ab5399f05ac614e1176442de0643c2e772ee148a" exitCode=0
Sep 14 11:22:29.812251 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.812153    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr" event={"ID":"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151","Type":"ContainerDied","Data":"bbf18ef2f38729a3d47e04c2ab5399f05ac614e1176442de0643c2e772ee148a"}
Sep 14 11:22:29.812251 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.812191    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr" event={"ID":"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151","Type":"ContainerDied","Data":"b7c12c24261aae188ea3bebe5b1b517932aa2c4e4aa186c95bed7c56d58ce624"}
Sep 14 11:22:29.812251 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.812207    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7c12c24261aae188ea3bebe5b1b517932aa2c4e4aa186c95bed7c56d58ce624"
Sep 14 11:22:29.822251 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.822194    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:22:29.910501 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.910471    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-openshift-service-ca\") pod \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") "
Sep 14 11:22:29.910501 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.910510    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6f66\" (UniqueName: \"kubernetes.io/projected/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-kube-api-access-c6f66\") pod \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") "
Sep 14 11:22:29.910774 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.910534    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-metrics-certs\") pod \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") "
Sep 14 11:22:29.910774 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.910595    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-maas-api-tls\") pod \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\" (UID: \"11f5c0bb-ed20-4eaa-a08c-621b2d2a9151\") "
Sep 14 11:22:29.910958 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.910928    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "11f5c0bb-ed20-4eaa-a08c-621b2d2a9151" (UID: "11f5c0bb-ed20-4eaa-a08c-621b2d2a9151"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:22:29.912668 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.912642    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "11f5c0bb-ed20-4eaa-a08c-621b2d2a9151" (UID: "11f5c0bb-ed20-4eaa-a08c-621b2d2a9151"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:22:29.912800 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.912696    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "11f5c0bb-ed20-4eaa-a08c-621b2d2a9151" (UID: "11f5c0bb-ed20-4eaa-a08c-621b2d2a9151"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:22:29.912800 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:29.912784    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-kube-api-access-c6f66" (OuterVolumeSpecName: "kube-api-access-c6f66") pod "11f5c0bb-ed20-4eaa-a08c-621b2d2a9151" (UID: "11f5c0bb-ed20-4eaa-a08c-621b2d2a9151"). InnerVolumeSpecName "kube-api-access-c6f66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:22:30.011478 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:30.011447    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:22:30.011478 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:30.011474    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6f66\" (UniqueName: \"kubernetes.io/projected/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-kube-api-access-c6f66\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:22:30.011478 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:30.011484    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:22:30.011752 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:30.011494    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:22:30.818216 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:30.818185    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"
Sep 14 11:22:30.844177 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:30.844152    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"]
Sep 14 11:22:30.850668 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:30.850639    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d5e418-68cb4f68dd-4l6vr"]
Sep 14 11:22:31.803331 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:31.803296    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f5c0bb-ed20-4eaa-a08c-621b2d2a9151" path="/var/lib/kubelet/pods/11f5c0bb-ed20-4eaa-a08c-621b2d2a9151/volumes"
Sep 14 11:22:34.116204 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:34.116165    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:22:54.537379 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:54.537289    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:22:56.996567 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:56.996530    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"]
Sep 14 11:22:56.996977 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:56.996822    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4" podUID="ca88e641-e8f3-4f23-9357-fc034752428b" containerName="maas-api" containerID="cri-o://12e041fdd6fc1f792d0ab5f9b55a9bbac24c7a202857430aa87177135ece8b62" gracePeriod=30
Sep 14 11:22:57.249014 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.248956    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:22:57.362466 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.362433    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca88e641-e8f3-4f23-9357-fc034752428b-metrics-certs\") pod \"ca88e641-e8f3-4f23-9357-fc034752428b\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") "
Sep 14 11:22:57.362657 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.362484    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca88e641-e8f3-4f23-9357-fc034752428b-maas-api-tls\") pod \"ca88e641-e8f3-4f23-9357-fc034752428b\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") "
Sep 14 11:22:57.362657 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.362507    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca88e641-e8f3-4f23-9357-fc034752428b-openshift-service-ca\") pod \"ca88e641-e8f3-4f23-9357-fc034752428b\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") "
Sep 14 11:22:57.362657 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.362566    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nhlk\" (UniqueName: \"kubernetes.io/projected/ca88e641-e8f3-4f23-9357-fc034752428b-kube-api-access-4nhlk\") pod \"ca88e641-e8f3-4f23-9357-fc034752428b\" (UID: \"ca88e641-e8f3-4f23-9357-fc034752428b\") "
Sep 14 11:22:57.363037 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.363005    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca88e641-e8f3-4f23-9357-fc034752428b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca88e641-e8f3-4f23-9357-fc034752428b" (UID: "ca88e641-e8f3-4f23-9357-fc034752428b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:22:57.364690 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.364660    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca88e641-e8f3-4f23-9357-fc034752428b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca88e641-e8f3-4f23-9357-fc034752428b" (UID: "ca88e641-e8f3-4f23-9357-fc034752428b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:22:57.364690 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.364682    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca88e641-e8f3-4f23-9357-fc034752428b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ca88e641-e8f3-4f23-9357-fc034752428b" (UID: "ca88e641-e8f3-4f23-9357-fc034752428b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:22:57.364874 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.364768    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca88e641-e8f3-4f23-9357-fc034752428b-kube-api-access-4nhlk" (OuterVolumeSpecName: "kube-api-access-4nhlk") pod "ca88e641-e8f3-4f23-9357-fc034752428b" (UID: "ca88e641-e8f3-4f23-9357-fc034752428b"). InnerVolumeSpecName "kube-api-access-4nhlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:22:57.463456 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.463424    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca88e641-e8f3-4f23-9357-fc034752428b-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:22:57.463456 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.463454    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca88e641-e8f3-4f23-9357-fc034752428b-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:22:57.463456 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.463463    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nhlk\" (UniqueName: \"kubernetes.io/projected/ca88e641-e8f3-4f23-9357-fc034752428b-kube-api-access-4nhlk\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:22:57.463683 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.463472    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca88e641-e8f3-4f23-9357-fc034752428b-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:22:57.978431 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.978399    2561 generic.go:356] "Generic (PLEG): container finished" podID="ca88e641-e8f3-4f23-9357-fc034752428b" containerID="12e041fdd6fc1f792d0ab5f9b55a9bbac24c7a202857430aa87177135ece8b62" exitCode=0
Sep 14 11:22:57.978583 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.978455    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"
Sep 14 11:22:57.978583 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.978482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4" event={"ID":"ca88e641-e8f3-4f23-9357-fc034752428b","Type":"ContainerDied","Data":"12e041fdd6fc1f792d0ab5f9b55a9bbac24c7a202857430aa87177135ece8b62"}
Sep 14 11:22:57.978583 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.978578    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4" event={"ID":"ca88e641-e8f3-4f23-9357-fc034752428b","Type":"ContainerDied","Data":"d5bec1bbe1ec3abc424002149bae08cd3ae98a1e76ceca4e2e01519d63753d94"}
Sep 14 11:22:57.978696 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.978605    2561 scope.go:117] "RemoveContainer" containerID="12e041fdd6fc1f792d0ab5f9b55a9bbac24c7a202857430aa87177135ece8b62"
Sep 14 11:22:57.988083 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.988067    2561 scope.go:117] "RemoveContainer" containerID="12e041fdd6fc1f792d0ab5f9b55a9bbac24c7a202857430aa87177135ece8b62"
Sep 14 11:22:57.988349 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:22:57.988327    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e041fdd6fc1f792d0ab5f9b55a9bbac24c7a202857430aa87177135ece8b62\": container with ID starting with 12e041fdd6fc1f792d0ab5f9b55a9bbac24c7a202857430aa87177135ece8b62 not found: ID does not exist" containerID="12e041fdd6fc1f792d0ab5f9b55a9bbac24c7a202857430aa87177135ece8b62"
Sep 14 11:22:57.988442 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:57.988357    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e041fdd6fc1f792d0ab5f9b55a9bbac24c7a202857430aa87177135ece8b62"} err="failed to get container status \"12e041fdd6fc1f792d0ab5f9b55a9bbac24c7a202857430aa87177135ece8b62\": rpc error: code = NotFound desc = could not find container \"12e041fdd6fc1f792d0ab5f9b55a9bbac24c7a202857430aa87177135ece8b62\": container with ID starting with 12e041fdd6fc1f792d0ab5f9b55a9bbac24c7a202857430aa87177135ece8b62 not found: ID does not exist"
Sep 14 11:22:58.016267 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:58.016229    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"]
Sep 14 11:22:58.032274 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:58.032241    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf45e0-6d85f6d468-98bq4"]
Sep 14 11:22:59.803267 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:22:59.803226    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca88e641-e8f3-4f23-9357-fc034752428b" path="/var/lib/kubelet/pods/ca88e641-e8f3-4f23-9357-fc034752428b/volumes"
Sep 14 11:23:06.818606 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:23:06.818574    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:23:16.005659 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:23:16.005632    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 11:23:16.012550 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:23:16.012526    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 11:24:23.237791 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.237701    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"]
Sep 14 11:24:23.238611 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.237943    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8" podUID="1b99baa9-b81b-446e-b4f3-e5fad2d6e693" containerName="maas-api" containerID="cri-o://c49ab3538e0514d9c8e2038795ca177f8f38ef38ba2a14267d997b05bd2d938b" gracePeriod=30
Sep 14 11:24:23.483107 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.483075    2561 generic.go:356] "Generic (PLEG): container finished" podID="1b99baa9-b81b-446e-b4f3-e5fad2d6e693" containerID="c49ab3538e0514d9c8e2038795ca177f8f38ef38ba2a14267d997b05bd2d938b" exitCode=0
Sep 14 11:24:23.483244 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.483156    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8" event={"ID":"1b99baa9-b81b-446e-b4f3-e5fad2d6e693","Type":"ContainerDied","Data":"c49ab3538e0514d9c8e2038795ca177f8f38ef38ba2a14267d997b05bd2d938b"}
Sep 14 11:24:23.483244 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.483220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8" event={"ID":"1b99baa9-b81b-446e-b4f3-e5fad2d6e693","Type":"ContainerDied","Data":"1c003970f69828cca2924184bcfd9791c0107246420672b0a057d26d89f0085c"}
Sep 14 11:24:23.483333 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.483243    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c003970f69828cca2924184bcfd9791c0107246420672b0a057d26d89f0085c"
Sep 14 11:24:23.495698 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.495645    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:24:23.631104 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.631074    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-metrics-certs\") pod \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") "
Sep 14 11:24:23.631258 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.631131    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-maas-api-tls\") pod \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") "
Sep 14 11:24:23.631258 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.631159    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-openshift-service-ca\") pod \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") "
Sep 14 11:24:23.631402 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.631381    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwpqb\" (UniqueName: \"kubernetes.io/projected/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-kube-api-access-cwpqb\") pod \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\" (UID: \"1b99baa9-b81b-446e-b4f3-e5fad2d6e693\") "
Sep 14 11:24:23.631541 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.631518    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b99baa9-b81b-446e-b4f3-e5fad2d6e693" (UID: "1b99baa9-b81b-446e-b4f3-e5fad2d6e693"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:24:23.631761 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.631740    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:23.633554 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.633519    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1b99baa9-b81b-446e-b4f3-e5fad2d6e693" (UID: "1b99baa9-b81b-446e-b4f3-e5fad2d6e693"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:24:23.633659 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.633554    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b99baa9-b81b-446e-b4f3-e5fad2d6e693" (UID: "1b99baa9-b81b-446e-b4f3-e5fad2d6e693"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:24:23.633752 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.633687    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-kube-api-access-cwpqb" (OuterVolumeSpecName: "kube-api-access-cwpqb") pod "1b99baa9-b81b-446e-b4f3-e5fad2d6e693" (UID: "1b99baa9-b81b-446e-b4f3-e5fad2d6e693"). InnerVolumeSpecName "kube-api-access-cwpqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:24:23.732618 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.732580    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwpqb\" (UniqueName: \"kubernetes.io/projected/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-kube-api-access-cwpqb\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:23.732618 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.732612    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:23.732618 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:23.732621    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b99baa9-b81b-446e-b4f3-e5fad2d6e693-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:24.489588 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:24.489551    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"
Sep 14 11:24:24.544526 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:24.544495    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"]
Sep 14 11:24:24.559345 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:24.559316    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a521b5-7b799ffb5c-wggd8"]
Sep 14 11:24:25.803064 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:25.803030    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b99baa9-b81b-446e-b4f3-e5fad2d6e693" path="/var/lib/kubelet/pods/1b99baa9-b81b-446e-b4f3-e5fad2d6e693/volumes"
Sep 14 11:24:26.567405 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:26.567218    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:24:31.573819 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:31.573735    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"]
Sep 14 11:24:31.574238 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:31.573924    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" podUID="196e2ccf-eb11-4841-b102-7e600293c333" containerName="istio-proxy" containerID="cri-o://37b79dfc52a24a1d9451d36cf594ea2766081f398a3cbba6123e03997a50c6da" gracePeriod=30
Sep 14 11:24:36.817183 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.817157    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:24:36.951345 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.951313    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-workload-certs\") pod \"196e2ccf-eb11-4841-b102-7e600293c333\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") "
Sep 14 11:24:36.951531 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.951362    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v99gf\" (UniqueName: \"kubernetes.io/projected/196e2ccf-eb11-4841-b102-7e600293c333-kube-api-access-v99gf\") pod \"196e2ccf-eb11-4841-b102-7e600293c333\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") "
Sep 14 11:24:36.951531 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.951391    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-workload-socket\") pod \"196e2ccf-eb11-4841-b102-7e600293c333\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") "
Sep 14 11:24:36.951531 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.951423    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196e2ccf-eb11-4841-b102-7e600293c333-istiod-ca-cert\") pod \"196e2ccf-eb11-4841-b102-7e600293c333\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") "
Sep 14 11:24:36.951531 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.951458    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196e2ccf-eb11-4841-b102-7e600293c333-istio-token\") pod \"196e2ccf-eb11-4841-b102-7e600293c333\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") "
Sep 14 11:24:36.951531 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.951486    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196e2ccf-eb11-4841-b102-7e600293c333-istio-podinfo\") pod \"196e2ccf-eb11-4841-b102-7e600293c333\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") "
Sep 14 11:24:36.951834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.951585    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-credential-socket\") pod \"196e2ccf-eb11-4841-b102-7e600293c333\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") "
Sep 14 11:24:36.951834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.951602    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "196e2ccf-eb11-4841-b102-7e600293c333" (UID: "196e2ccf-eb11-4841-b102-7e600293c333"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:36.951834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.951638    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-istio-data\") pod \"196e2ccf-eb11-4841-b102-7e600293c333\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") "
Sep 14 11:24:36.951834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.951667    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-istio-envoy\") pod \"196e2ccf-eb11-4841-b102-7e600293c333\" (UID: \"196e2ccf-eb11-4841-b102-7e600293c333\") "
Sep 14 11:24:36.951834 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.951668    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "196e2ccf-eb11-4841-b102-7e600293c333" (UID: "196e2ccf-eb11-4841-b102-7e600293c333"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:36.952068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.951886    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "196e2ccf-eb11-4841-b102-7e600293c333" (UID: "196e2ccf-eb11-4841-b102-7e600293c333"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:36.952068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.951914    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-istio-data" (OuterVolumeSpecName: "istio-data") pod "196e2ccf-eb11-4841-b102-7e600293c333" (UID: "196e2ccf-eb11-4841-b102-7e600293c333"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:36.952068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.952033    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-istio-data\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:36.952068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.952050    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-workload-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:36.952068 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.952064    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-workload-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:36.952234 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.952076    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-credential-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:36.952269 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.952245    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196e2ccf-eb11-4841-b102-7e600293c333-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "196e2ccf-eb11-4841-b102-7e600293c333" (UID: "196e2ccf-eb11-4841-b102-7e600293c333"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:24:36.953867 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.953840    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196e2ccf-eb11-4841-b102-7e600293c333-istio-token" (OuterVolumeSpecName: "istio-token") pod "196e2ccf-eb11-4841-b102-7e600293c333" (UID: "196e2ccf-eb11-4841-b102-7e600293c333"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:24:36.953973 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.953862    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/196e2ccf-eb11-4841-b102-7e600293c333-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "196e2ccf-eb11-4841-b102-7e600293c333" (UID: "196e2ccf-eb11-4841-b102-7e600293c333"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 11:24:36.954065 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.954046    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "196e2ccf-eb11-4841-b102-7e600293c333" (UID: "196e2ccf-eb11-4841-b102-7e600293c333"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:36.954124 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:36.954052    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196e2ccf-eb11-4841-b102-7e600293c333-kube-api-access-v99gf" (OuterVolumeSpecName: "kube-api-access-v99gf") pod "196e2ccf-eb11-4841-b102-7e600293c333" (UID: "196e2ccf-eb11-4841-b102-7e600293c333"). InnerVolumeSpecName "kube-api-access-v99gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:24:37.053330 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.053303    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196e2ccf-eb11-4841-b102-7e600293c333-istio-envoy\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:37.053330 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.053330    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v99gf\" (UniqueName: \"kubernetes.io/projected/196e2ccf-eb11-4841-b102-7e600293c333-kube-api-access-v99gf\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:37.053524 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.053339    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196e2ccf-eb11-4841-b102-7e600293c333-istiod-ca-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:37.053524 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.053348    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196e2ccf-eb11-4841-b102-7e600293c333-istio-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:37.053524 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.053357    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196e2ccf-eb11-4841-b102-7e600293c333-istio-podinfo\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:37.569348 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.569320    2561 generic.go:356] "Generic (PLEG): container finished" podID="196e2ccf-eb11-4841-b102-7e600293c333" containerID="37b79dfc52a24a1d9451d36cf594ea2766081f398a3cbba6123e03997a50c6da" exitCode=0
Sep 14 11:24:37.569522 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.569376    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" event={"ID":"196e2ccf-eb11-4841-b102-7e600293c333","Type":"ContainerDied","Data":"37b79dfc52a24a1d9451d36cf594ea2766081f398a3cbba6123e03997a50c6da"}
Sep 14 11:24:37.569522 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.569391    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"
Sep 14 11:24:37.569522 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.569422    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j" event={"ID":"196e2ccf-eb11-4841-b102-7e600293c333","Type":"ContainerDied","Data":"0f332624539fd34865bf98d0fb71653792cba2cc6f99c1f07ab891c759c31c3b"}
Sep 14 11:24:37.569522 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.569445    2561 scope.go:117] "RemoveContainer" containerID="37b79dfc52a24a1d9451d36cf594ea2766081f398a3cbba6123e03997a50c6da"
Sep 14 11:24:37.579577 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.579559    2561 scope.go:117] "RemoveContainer" containerID="650900a9f666adea08b7f2fdc685646b60c3526e4963724b25ee9c0e81a0dd1d"
Sep 14 11:24:37.587677 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.587658    2561 scope.go:117] "RemoveContainer" containerID="37b79dfc52a24a1d9451d36cf594ea2766081f398a3cbba6123e03997a50c6da"
Sep 14 11:24:37.587923 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:24:37.587895    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b79dfc52a24a1d9451d36cf594ea2766081f398a3cbba6123e03997a50c6da\": container with ID starting with 37b79dfc52a24a1d9451d36cf594ea2766081f398a3cbba6123e03997a50c6da not found: ID does not exist" containerID="37b79dfc52a24a1d9451d36cf594ea2766081f398a3cbba6123e03997a50c6da"
Sep 14 11:24:37.587982 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.587931    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b79dfc52a24a1d9451d36cf594ea2766081f398a3cbba6123e03997a50c6da"} err="failed to get container status \"37b79dfc52a24a1d9451d36cf594ea2766081f398a3cbba6123e03997a50c6da\": rpc error: code = NotFound desc = could not find container \"37b79dfc52a24a1d9451d36cf594ea2766081f398a3cbba6123e03997a50c6da\": container with ID starting with 37b79dfc52a24a1d9451d36cf594ea2766081f398a3cbba6123e03997a50c6da not found: ID does not exist"
Sep 14 11:24:37.587982 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.587955    2561 scope.go:117] "RemoveContainer" containerID="650900a9f666adea08b7f2fdc685646b60c3526e4963724b25ee9c0e81a0dd1d"
Sep 14 11:24:37.588158 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:24:37.588142    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650900a9f666adea08b7f2fdc685646b60c3526e4963724b25ee9c0e81a0dd1d\": container with ID starting with 650900a9f666adea08b7f2fdc685646b60c3526e4963724b25ee9c0e81a0dd1d not found: ID does not exist" containerID="650900a9f666adea08b7f2fdc685646b60c3526e4963724b25ee9c0e81a0dd1d"
Sep 14 11:24:37.588205 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.588162    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650900a9f666adea08b7f2fdc685646b60c3526e4963724b25ee9c0e81a0dd1d"} err="failed to get container status \"650900a9f666adea08b7f2fdc685646b60c3526e4963724b25ee9c0e81a0dd1d\": rpc error: code = NotFound desc = could not find container \"650900a9f666adea08b7f2fdc685646b60c3526e4963724b25ee9c0e81a0dd1d\": container with ID starting with 650900a9f666adea08b7f2fdc685646b60c3526e4963724b25ee9c0e81a0dd1d not found: ID does not exist"
Sep 14 11:24:37.597137 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.597112    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"]
Sep 14 11:24:37.599427 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.599407    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a521b5-openshift-default-8486f85b48-sdb5j"]
Sep 14 11:24:37.803335 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:37.803302    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="196e2ccf-eb11-4841-b102-7e600293c333" path="/var/lib/kubelet/pods/196e2ccf-eb11-4841-b102-7e600293c333/volumes"
Sep 14 11:24:42.600639 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:42.600608    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"]
Sep 14 11:24:42.601263 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:42.600852    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv" podUID="88d570b9-8384-4305-95b1-a2326de4100a" containerName="maas-api" containerID="cri-o://a7d9e93c3bc37fdae139ad359e190fd6847b54d36ee8051376d9e43e1db5f574" gracePeriod=30
Sep 14 11:24:42.853756 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:42.853682    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:24:43.004575 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.004544    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-maas-api-tls\") pod \"88d570b9-8384-4305-95b1-a2326de4100a\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") "
Sep 14 11:24:43.004821 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.004644    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2tr6\" (UniqueName: \"kubernetes.io/projected/88d570b9-8384-4305-95b1-a2326de4100a-kube-api-access-l2tr6\") pod \"88d570b9-8384-4305-95b1-a2326de4100a\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") "
Sep 14 11:24:43.004821 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.004666    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-metrics-certs\") pod \"88d570b9-8384-4305-95b1-a2326de4100a\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") "
Sep 14 11:24:43.004821 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.004753    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d570b9-8384-4305-95b1-a2326de4100a-openshift-service-ca\") pod \"88d570b9-8384-4305-95b1-a2326de4100a\" (UID: \"88d570b9-8384-4305-95b1-a2326de4100a\") "
Sep 14 11:24:43.005330 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.005285    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d570b9-8384-4305-95b1-a2326de4100a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "88d570b9-8384-4305-95b1-a2326de4100a" (UID: "88d570b9-8384-4305-95b1-a2326de4100a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:24:43.007318 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.007279    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "88d570b9-8384-4305-95b1-a2326de4100a" (UID: "88d570b9-8384-4305-95b1-a2326de4100a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:24:43.007456 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.007350    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "88d570b9-8384-4305-95b1-a2326de4100a" (UID: "88d570b9-8384-4305-95b1-a2326de4100a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:24:43.007456 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.007438    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d570b9-8384-4305-95b1-a2326de4100a-kube-api-access-l2tr6" (OuterVolumeSpecName: "kube-api-access-l2tr6") pod "88d570b9-8384-4305-95b1-a2326de4100a" (UID: "88d570b9-8384-4305-95b1-a2326de4100a"). InnerVolumeSpecName "kube-api-access-l2tr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:24:43.106067 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.106033    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d570b9-8384-4305-95b1-a2326de4100a-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:43.106067 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.106069    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:43.106258 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.106079    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2tr6\" (UniqueName: \"kubernetes.io/projected/88d570b9-8384-4305-95b1-a2326de4100a-kube-api-access-l2tr6\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:43.106258 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.106090    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d570b9-8384-4305-95b1-a2326de4100a-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:43.605250 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.605223    2561 generic.go:356] "Generic (PLEG): container finished" podID="88d570b9-8384-4305-95b1-a2326de4100a" containerID="a7d9e93c3bc37fdae139ad359e190fd6847b54d36ee8051376d9e43e1db5f574" exitCode=0
Sep 14 11:24:43.605689 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.605260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv" event={"ID":"88d570b9-8384-4305-95b1-a2326de4100a","Type":"ContainerDied","Data":"a7d9e93c3bc37fdae139ad359e190fd6847b54d36ee8051376d9e43e1db5f574"}
Sep 14 11:24:43.605689 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.605268    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"
Sep 14 11:24:43.605689 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.605297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv" event={"ID":"88d570b9-8384-4305-95b1-a2326de4100a","Type":"ContainerDied","Data":"f84fd11c63ecdfb942a1d2df68337e4becc448a2fb6f92b248b05a087e5a2c75"}
Sep 14 11:24:43.605689 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.605319    2561 scope.go:117] "RemoveContainer" containerID="a7d9e93c3bc37fdae139ad359e190fd6847b54d36ee8051376d9e43e1db5f574"
Sep 14 11:24:43.614995 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.614976    2561 scope.go:117] "RemoveContainer" containerID="a7d9e93c3bc37fdae139ad359e190fd6847b54d36ee8051376d9e43e1db5f574"
Sep 14 11:24:43.615294 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:24:43.615269    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d9e93c3bc37fdae139ad359e190fd6847b54d36ee8051376d9e43e1db5f574\": container with ID starting with a7d9e93c3bc37fdae139ad359e190fd6847b54d36ee8051376d9e43e1db5f574 not found: ID does not exist" containerID="a7d9e93c3bc37fdae139ad359e190fd6847b54d36ee8051376d9e43e1db5f574"
Sep 14 11:24:43.615363 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.615296    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d9e93c3bc37fdae139ad359e190fd6847b54d36ee8051376d9e43e1db5f574"} err="failed to get container status \"a7d9e93c3bc37fdae139ad359e190fd6847b54d36ee8051376d9e43e1db5f574\": rpc error: code = NotFound desc = could not find container \"a7d9e93c3bc37fdae139ad359e190fd6847b54d36ee8051376d9e43e1db5f574\": container with ID starting with a7d9e93c3bc37fdae139ad359e190fd6847b54d36ee8051376d9e43e1db5f574 not found: ID does not exist"
Sep 14 11:24:43.631521 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.631497    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"]
Sep 14 11:24:43.635470 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.635450    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdbc00-7b474c8d6d-cbmmv"]
Sep 14 11:24:43.803240 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:43.803200    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d570b9-8384-4305-95b1-a2326de4100a" path="/var/lib/kubelet/pods/88d570b9-8384-4305-95b1-a2326de4100a/volumes"
Sep 14 11:24:45.418032 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:45.417995    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:24:50.103945 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:50.103905    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"]
Sep 14 11:24:50.104310 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:50.104120    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy" containerID="cri-o://a80414b796e0fcdcba2c1bb3b7536122a3c05827f3b88f7a97dcffc7c47e318f" gracePeriod=30
Sep 14 11:24:50.941708 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:50.941664    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 11:24:50.941897 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:50.941757    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 11:24:55.358599 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.358569    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:24:55.424682 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.424614    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-envoy\") pod \"5f557605-f940-4ae6-9054-3dc0cee19d46\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") "
Sep 14 11:24:55.424682 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.424676    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlz6j\" (UniqueName: \"kubernetes.io/projected/5f557605-f940-4ae6-9054-3dc0cee19d46-kube-api-access-dlz6j\") pod \"5f557605-f940-4ae6-9054-3dc0cee19d46\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") "
Sep 14 11:24:55.424854 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.424810    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-data\") pod \"5f557605-f940-4ae6-9054-3dc0cee19d46\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") "
Sep 14 11:24:55.424854 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.424845    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f557605-f940-4ae6-9054-3dc0cee19d46-istiod-ca-cert\") pod \"5f557605-f940-4ae6-9054-3dc0cee19d46\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") "
Sep 14 11:24:55.424944 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.424880    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-podinfo\") pod \"5f557605-f940-4ae6-9054-3dc0cee19d46\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") "
Sep 14 11:24:55.424944 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.424933    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-workload-certs\") pod \"5f557605-f940-4ae6-9054-3dc0cee19d46\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") "
Sep 14 11:24:55.425050 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.424972    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-workload-socket\") pod \"5f557605-f940-4ae6-9054-3dc0cee19d46\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") "
Sep 14 11:24:55.425104 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.425049    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f557605-f940-4ae6-9054-3dc0cee19d46" (UID: "5f557605-f940-4ae6-9054-3dc0cee19d46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:55.425236 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.425196    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f557605-f940-4ae6-9054-3dc0cee19d46" (UID: "5f557605-f940-4ae6-9054-3dc0cee19d46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:55.425311 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.425263    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-credential-socket\") pod \"5f557605-f940-4ae6-9054-3dc0cee19d46\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") "
Sep 14 11:24:55.425311 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.425272    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f557605-f940-4ae6-9054-3dc0cee19d46" (UID: "5f557605-f940-4ae6-9054-3dc0cee19d46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:55.425311 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.425283    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f557605-f940-4ae6-9054-3dc0cee19d46-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f557605-f940-4ae6-9054-3dc0cee19d46" (UID: "5f557605-f940-4ae6-9054-3dc0cee19d46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:24:55.425475 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.425331    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-token\") pod \"5f557605-f940-4ae6-9054-3dc0cee19d46\" (UID: \"5f557605-f940-4ae6-9054-3dc0cee19d46\") "
Sep 14 11:24:55.425475 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.425459    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f557605-f940-4ae6-9054-3dc0cee19d46" (UID: "5f557605-f940-4ae6-9054-3dc0cee19d46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:55.425791 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.425770    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-data\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:55.425908 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.425797    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f557605-f940-4ae6-9054-3dc0cee19d46-istiod-ca-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:55.425908 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.425816    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-workload-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:55.425908 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.425829    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-workload-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:55.425908 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.425841    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-credential-socket\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:55.426873 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.426841    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f557605-f940-4ae6-9054-3dc0cee19d46-kube-api-access-dlz6j" (OuterVolumeSpecName: "kube-api-access-dlz6j") pod "5f557605-f940-4ae6-9054-3dc0cee19d46" (UID: "5f557605-f940-4ae6-9054-3dc0cee19d46"). InnerVolumeSpecName "kube-api-access-dlz6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:24:55.426977 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.426954    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f557605-f940-4ae6-9054-3dc0cee19d46" (UID: "5f557605-f940-4ae6-9054-3dc0cee19d46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:55.427158 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.427131    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f557605-f940-4ae6-9054-3dc0cee19d46" (UID: "5f557605-f940-4ae6-9054-3dc0cee19d46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 11:24:55.427280 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.427264    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f557605-f940-4ae6-9054-3dc0cee19d46" (UID: "5f557605-f940-4ae6-9054-3dc0cee19d46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:24:55.526594 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.526560    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:55.526594 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.526587    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-envoy\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:55.526594 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.526600    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlz6j\" (UniqueName: \"kubernetes.io/projected/5f557605-f940-4ae6-9054-3dc0cee19d46-kube-api-access-dlz6j\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:55.526835 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.526614    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f557605-f940-4ae6-9054-3dc0cee19d46-istio-podinfo\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:24:55.677003 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.676911    2561 generic.go:356] "Generic (PLEG): container finished" podID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerID="a80414b796e0fcdcba2c1bb3b7536122a3c05827f3b88f7a97dcffc7c47e318f" exitCode=0
Sep 14 11:24:55.677003 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.676990    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"
Sep 14 11:24:55.677003 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.676989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" event={"ID":"5f557605-f940-4ae6-9054-3dc0cee19d46","Type":"ContainerDied","Data":"a80414b796e0fcdcba2c1bb3b7536122a3c05827f3b88f7a97dcffc7c47e318f"}
Sep 14 11:24:55.677245 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.677058    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk" event={"ID":"5f557605-f940-4ae6-9054-3dc0cee19d46","Type":"ContainerDied","Data":"3f0c2cc2cbcac5a65be1e3973f8fb96b69f6ba423ae22cf5577316d76c00babf"}
Sep 14 11:24:55.677245 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.677089    2561 scope.go:117] "RemoveContainer" containerID="a80414b796e0fcdcba2c1bb3b7536122a3c05827f3b88f7a97dcffc7c47e318f"
Sep 14 11:24:55.692434 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.692414    2561 scope.go:117] "RemoveContainer" containerID="6bafa7e4d7ccb808fb9e5df059516243770b2ce3b4bed7b11c6d115ee265e90c"
Sep 14 11:24:55.700939 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.700919    2561 scope.go:117] "RemoveContainer" containerID="a80414b796e0fcdcba2c1bb3b7536122a3c05827f3b88f7a97dcffc7c47e318f"
Sep 14 11:24:55.701210 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:24:55.701192    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80414b796e0fcdcba2c1bb3b7536122a3c05827f3b88f7a97dcffc7c47e318f\": container with ID starting with a80414b796e0fcdcba2c1bb3b7536122a3c05827f3b88f7a97dcffc7c47e318f not found: ID does not exist" containerID="a80414b796e0fcdcba2c1bb3b7536122a3c05827f3b88f7a97dcffc7c47e318f"
Sep 14 11:24:55.701280 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.701216    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80414b796e0fcdcba2c1bb3b7536122a3c05827f3b88f7a97dcffc7c47e318f"} err="failed to get container status \"a80414b796e0fcdcba2c1bb3b7536122a3c05827f3b88f7a97dcffc7c47e318f\": rpc error: code = NotFound desc = could not find container \"a80414b796e0fcdcba2c1bb3b7536122a3c05827f3b88f7a97dcffc7c47e318f\": container with ID starting with a80414b796e0fcdcba2c1bb3b7536122a3c05827f3b88f7a97dcffc7c47e318f not found: ID does not exist"
Sep 14 11:24:55.701280 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.701232    2561 scope.go:117] "RemoveContainer" containerID="6bafa7e4d7ccb808fb9e5df059516243770b2ce3b4bed7b11c6d115ee265e90c"
Sep 14 11:24:55.701459 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:24:55.701440    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bafa7e4d7ccb808fb9e5df059516243770b2ce3b4bed7b11c6d115ee265e90c\": container with ID starting with 6bafa7e4d7ccb808fb9e5df059516243770b2ce3b4bed7b11c6d115ee265e90c not found: ID does not exist" containerID="6bafa7e4d7ccb808fb9e5df059516243770b2ce3b4bed7b11c6d115ee265e90c"
Sep 14 11:24:55.701498 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.701471    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bafa7e4d7ccb808fb9e5df059516243770b2ce3b4bed7b11c6d115ee265e90c"} err="failed to get container status \"6bafa7e4d7ccb808fb9e5df059516243770b2ce3b4bed7b11c6d115ee265e90c\": rpc error: code = NotFound desc = could not find container \"6bafa7e4d7ccb808fb9e5df059516243770b2ce3b4bed7b11c6d115ee265e90c\": container with ID starting with 6bafa7e4d7ccb808fb9e5df059516243770b2ce3b4bed7b11c6d115ee265e90c not found: ID does not exist"
Sep 14 11:24:55.713102 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.713076    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"]
Sep 14 11:24:55.716363 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.716343    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fdbc00-openshift-default-86965b886b-s92bk"]
Sep 14 11:24:55.802864 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:24:55.802839    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" path="/var/lib/kubelet/pods/5f557605-f940-4ae6-9054-3dc0cee19d46/volumes"
Sep 14 11:25:01.276475 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.276442    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"]
Sep 14 11:25:01.280957 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.277139    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j" podUID="e6f445db-c180-4d42-838e-11d7fe0fc0ad" containerName="maas-api" containerID="cri-o://d89454fbfe9cf589d597c7ad25c20fe444271d41265286182e1b7f0095b3efed" gracePeriod=30
Sep 14 11:25:01.536919 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.536858    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:25:01.576023 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.575981    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mrv2\" (UniqueName: \"kubernetes.io/projected/e6f445db-c180-4d42-838e-11d7fe0fc0ad-kube-api-access-4mrv2\") pod \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") "
Sep 14 11:25:01.576209 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.576035    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-metrics-certs\") pod \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") "
Sep 14 11:25:01.576209 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.576080    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f445db-c180-4d42-838e-11d7fe0fc0ad-openshift-service-ca\") pod \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") "
Sep 14 11:25:01.576209 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.576195    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-maas-api-tls\") pod \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\" (UID: \"e6f445db-c180-4d42-838e-11d7fe0fc0ad\") "
Sep 14 11:25:01.576830 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.576775    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f445db-c180-4d42-838e-11d7fe0fc0ad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e6f445db-c180-4d42-838e-11d7fe0fc0ad" (UID: "e6f445db-c180-4d42-838e-11d7fe0fc0ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:25:01.578831 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.578797    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f445db-c180-4d42-838e-11d7fe0fc0ad-kube-api-access-4mrv2" (OuterVolumeSpecName: "kube-api-access-4mrv2") pod "e6f445db-c180-4d42-838e-11d7fe0fc0ad" (UID: "e6f445db-c180-4d42-838e-11d7fe0fc0ad"). InnerVolumeSpecName "kube-api-access-4mrv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:25:01.578959 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.578836    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e6f445db-c180-4d42-838e-11d7fe0fc0ad" (UID: "e6f445db-c180-4d42-838e-11d7fe0fc0ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:25:01.579380 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.579342    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e6f445db-c180-4d42-838e-11d7fe0fc0ad" (UID: "e6f445db-c180-4d42-838e-11d7fe0fc0ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:25:01.677090 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.677052    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mrv2\" (UniqueName: \"kubernetes.io/projected/e6f445db-c180-4d42-838e-11d7fe0fc0ad-kube-api-access-4mrv2\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:25:01.677090 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.677087    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:25:01.677090 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.677097    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f445db-c180-4d42-838e-11d7fe0fc0ad-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:25:01.677320 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.677106    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6f445db-c180-4d42-838e-11d7fe0fc0ad-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:25:01.714365 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.714324    2561 generic.go:356] "Generic (PLEG): container finished" podID="e6f445db-c180-4d42-838e-11d7fe0fc0ad" containerID="d89454fbfe9cf589d597c7ad25c20fe444271d41265286182e1b7f0095b3efed" exitCode=0
Sep 14 11:25:01.714523 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.714381    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"
Sep 14 11:25:01.714523 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.714408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j" event={"ID":"e6f445db-c180-4d42-838e-11d7fe0fc0ad","Type":"ContainerDied","Data":"d89454fbfe9cf589d597c7ad25c20fe444271d41265286182e1b7f0095b3efed"}
Sep 14 11:25:01.714523 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.714495    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j" event={"ID":"e6f445db-c180-4d42-838e-11d7fe0fc0ad","Type":"ContainerDied","Data":"0e06646b555bb2b2d3a935dc6d55be9f8f96c91a739e8ca5d4d827066605b3c4"}
Sep 14 11:25:01.714523 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.714522    2561 scope.go:117] "RemoveContainer" containerID="d89454fbfe9cf589d597c7ad25c20fe444271d41265286182e1b7f0095b3efed"
Sep 14 11:25:01.724554 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.724536    2561 scope.go:117] "RemoveContainer" containerID="d89454fbfe9cf589d597c7ad25c20fe444271d41265286182e1b7f0095b3efed"
Sep 14 11:25:01.724833 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:25:01.724814    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89454fbfe9cf589d597c7ad25c20fe444271d41265286182e1b7f0095b3efed\": container with ID starting with d89454fbfe9cf589d597c7ad25c20fe444271d41265286182e1b7f0095b3efed not found: ID does not exist" containerID="d89454fbfe9cf589d597c7ad25c20fe444271d41265286182e1b7f0095b3efed"
Sep 14 11:25:01.724892 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.724841    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89454fbfe9cf589d597c7ad25c20fe444271d41265286182e1b7f0095b3efed"} err="failed to get container status \"d89454fbfe9cf589d597c7ad25c20fe444271d41265286182e1b7f0095b3efed\": rpc error: code = NotFound desc = could not find container \"d89454fbfe9cf589d597c7ad25c20fe444271d41265286182e1b7f0095b3efed\": container with ID starting with d89454fbfe9cf589d597c7ad25c20fe444271d41265286182e1b7f0095b3efed not found: ID does not exist"
Sep 14 11:25:01.746356 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.746328    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"]
Sep 14 11:25:01.753355 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.753334    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0fb86d-db6647f6b-d8f6j"]
Sep 14 11:25:01.804883 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:01.804805    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6f445db-c180-4d42-838e-11d7fe0fc0ad" path="/var/lib/kubelet/pods/e6f445db-c180-4d42-838e-11d7fe0fc0ad/volumes"
Sep 14 11:25:04.021367 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:04.021336    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:25:13.009849 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:25:13.009811    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:26:12.972796 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.972758    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-785cb4f769-52kcw"]
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973120    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f5c0bb-ed20-4eaa-a08c-621b2d2a9151" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973131    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f5c0bb-ed20-4eaa-a08c-621b2d2a9151" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973141    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973146    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973156    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="196e2ccf-eb11-4841-b102-7e600293c333" containerName="istio-proxy"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973162    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="196e2ccf-eb11-4841-b102-7e600293c333" containerName="istio-proxy"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973173    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca88e641-e8f3-4f23-9357-fc034752428b" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973178    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca88e641-e8f3-4f23-9357-fc034752428b" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973193    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d570b9-8384-4305-95b1-a2326de4100a" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973199    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d570b9-8384-4305-95b1-a2326de4100a" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973205    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6f445db-c180-4d42-838e-11d7fe0fc0ad" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973212    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6f445db-c180-4d42-838e-11d7fe0fc0ad" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973218    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b99baa9-b81b-446e-b4f3-e5fad2d6e693" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973223    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b99baa9-b81b-446e-b4f3-e5fad2d6e693" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973274    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973283    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b99baa9-b81b-446e-b4f3-e5fad2d6e693" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973289    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973294    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f5c0bb-ed20-4eaa-a08c-621b2d2a9151" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973299    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="196e2ccf-eb11-4841-b102-7e600293c333" containerName="istio-proxy"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973304    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d570b9-8384-4305-95b1-a2326de4100a" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973309    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6f445db-c180-4d42-838e-11d7fe0fc0ad" containerName="maas-api"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973315    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="196e2ccf-eb11-4841-b102-7e600293c333" containerName="istio-proxy"
Sep 14 11:26:12.975181 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.973322    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca88e641-e8f3-4f23-9357-fc034752428b" containerName="maas-api"
Sep 14 11:26:12.976133 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.976116    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:12.980011 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.979983    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 14 11:26:12.980011 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.980002    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 14 11:26:12.980195 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.980017    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lv5gj\""
Sep 14 11:26:12.980195 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.980023    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 14 11:26:12.986416 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:12.986392    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-785cb4f769-52kcw"]
Sep 14 11:26:13.089117 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:13.089068    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-metrics-certs\") pod \"maas-controller-785cb4f769-52kcw\" (UID: \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\") " pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:13.089117 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:13.089121    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xq2\" (UniqueName: \"kubernetes.io/projected/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-kube-api-access-g7xq2\") pod \"maas-controller-785cb4f769-52kcw\" (UID: \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\") " pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:13.089334 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:13.089265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-webhook-cert\") pod \"maas-controller-785cb4f769-52kcw\" (UID: \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\") " pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:13.190616 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:13.190591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-webhook-cert\") pod \"maas-controller-785cb4f769-52kcw\" (UID: \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\") " pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:13.190830 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:13.190637    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-metrics-certs\") pod \"maas-controller-785cb4f769-52kcw\" (UID: \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\") " pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:13.190830 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:13.190656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xq2\" (UniqueName: \"kubernetes.io/projected/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-kube-api-access-g7xq2\") pod \"maas-controller-785cb4f769-52kcw\" (UID: \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\") " pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:13.193126 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:13.193101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-metrics-certs\") pod \"maas-controller-785cb4f769-52kcw\" (UID: \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\") " pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:13.193225 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:13.193178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-webhook-cert\") pod \"maas-controller-785cb4f769-52kcw\" (UID: \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\") " pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:13.199313 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:13.199288    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xq2\" (UniqueName: \"kubernetes.io/projected/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-kube-api-access-g7xq2\") pod \"maas-controller-785cb4f769-52kcw\" (UID: \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\") " pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:13.287217 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:13.287156    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:13.562179 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:13.562156    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-785cb4f769-52kcw"]
Sep 14 11:26:13.563605 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:26:13.563578    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9e26e5_5c8f_4b62_bc93_0be0888e371f.slice/crio-18cb6c7f78621b66e9ca781d7029892c142df4d971e54011194ea4fff77ee0e4 WatchSource:0}: Error finding container 18cb6c7f78621b66e9ca781d7029892c142df4d971e54011194ea4fff77ee0e4: Status 404 returned error can't find the container with id 18cb6c7f78621b66e9ca781d7029892c142df4d971e54011194ea4fff77ee0e4
Sep 14 11:26:13.564963 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:13.564944    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 11:26:14.117184 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:14.117152    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785cb4f769-52kcw" event={"ID":"cd9e26e5-5c8f-4b62-bc93-0be0888e371f","Type":"ContainerStarted","Data":"18cb6c7f78621b66e9ca781d7029892c142df4d971e54011194ea4fff77ee0e4"}
Sep 14 11:26:16.131915 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:16.131879    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785cb4f769-52kcw" event={"ID":"cd9e26e5-5c8f-4b62-bc93-0be0888e371f","Type":"ContainerStarted","Data":"219c9c04bfca53972ebe8b32b677f803bd87f967203339d18ed77e22c574781c"}
Sep 14 11:26:16.132426 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:16.131938    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:16.151629 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:16.151578    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-785cb4f769-52kcw" podStartSLOduration=2.348675448 podStartE2EDuration="4.15156132s" totalImagesPullingTime="1.802885872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:26:16.149699599 +0000 UTC m=+2280.834383989" watchObservedRunningTime="2026-09-14 11:26:16.15156132 +0000 UTC m=+2280.836245703"
Sep 14 11:26:27.139155 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:27.139124    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:38.117873 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:38.117835    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:26:38.551546 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:38.551452    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-785cb4f769-52kcw"]
Sep 14 11:26:38.551739 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:38.551692    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-785cb4f769-52kcw" podUID="cd9e26e5-5c8f-4b62-bc93-0be0888e371f" containerName="manager" containerID="cri-o://219c9c04bfca53972ebe8b32b677f803bd87f967203339d18ed77e22c574781c" gracePeriod=30
Sep 14 11:26:38.798379 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:38.798355    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:38.901403 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:38.901377    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-webhook-cert\") pod \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\" (UID: \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\") "
Sep 14 11:26:38.901571 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:38.901466    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-metrics-certs\") pod \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\" (UID: \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\") "
Sep 14 11:26:38.901644 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:38.901607    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7xq2\" (UniqueName: \"kubernetes.io/projected/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-kube-api-access-g7xq2\") pod \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\" (UID: \"cd9e26e5-5c8f-4b62-bc93-0be0888e371f\") "
Sep 14 11:26:38.903405 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:38.903375    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cd9e26e5-5c8f-4b62-bc93-0be0888e371f" (UID: "cd9e26e5-5c8f-4b62-bc93-0be0888e371f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:26:38.903563 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:38.903545    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-kube-api-access-g7xq2" (OuterVolumeSpecName: "kube-api-access-g7xq2") pod "cd9e26e5-5c8f-4b62-bc93-0be0888e371f" (UID: "cd9e26e5-5c8f-4b62-bc93-0be0888e371f"). InnerVolumeSpecName "kube-api-access-g7xq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:26:38.903563 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:38.903552    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "cd9e26e5-5c8f-4b62-bc93-0be0888e371f" (UID: "cd9e26e5-5c8f-4b62-bc93-0be0888e371f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:26:39.002731 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:39.002690    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-webhook-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:26:39.002731 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:39.002724    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:26:39.002731 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:39.002734    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7xq2\" (UniqueName: \"kubernetes.io/projected/cd9e26e5-5c8f-4b62-bc93-0be0888e371f-kube-api-access-g7xq2\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:26:39.266479 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:39.266393    2561 generic.go:356] "Generic (PLEG): container finished" podID="cd9e26e5-5c8f-4b62-bc93-0be0888e371f" containerID="219c9c04bfca53972ebe8b32b677f803bd87f967203339d18ed77e22c574781c" exitCode=0
Sep 14 11:26:39.266479 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:39.266436    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785cb4f769-52kcw"
Sep 14 11:26:39.267014 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:39.266486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785cb4f769-52kcw" event={"ID":"cd9e26e5-5c8f-4b62-bc93-0be0888e371f","Type":"ContainerDied","Data":"219c9c04bfca53972ebe8b32b677f803bd87f967203339d18ed77e22c574781c"}
Sep 14 11:26:39.267014 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:39.266546    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785cb4f769-52kcw" event={"ID":"cd9e26e5-5c8f-4b62-bc93-0be0888e371f","Type":"ContainerDied","Data":"18cb6c7f78621b66e9ca781d7029892c142df4d971e54011194ea4fff77ee0e4"}
Sep 14 11:26:39.267014 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:39.266580    2561 scope.go:117] "RemoveContainer" containerID="219c9c04bfca53972ebe8b32b677f803bd87f967203339d18ed77e22c574781c"
Sep 14 11:26:39.276273 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:39.276257    2561 scope.go:117] "RemoveContainer" containerID="219c9c04bfca53972ebe8b32b677f803bd87f967203339d18ed77e22c574781c"
Sep 14 11:26:39.276513 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:26:39.276494    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219c9c04bfca53972ebe8b32b677f803bd87f967203339d18ed77e22c574781c\": container with ID starting with 219c9c04bfca53972ebe8b32b677f803bd87f967203339d18ed77e22c574781c not found: ID does not exist" containerID="219c9c04bfca53972ebe8b32b677f803bd87f967203339d18ed77e22c574781c"
Sep 14 11:26:39.276562 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:39.276521    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219c9c04bfca53972ebe8b32b677f803bd87f967203339d18ed77e22c574781c"} err="failed to get container status \"219c9c04bfca53972ebe8b32b677f803bd87f967203339d18ed77e22c574781c\": rpc error: code = NotFound desc = could not find container \"219c9c04bfca53972ebe8b32b677f803bd87f967203339d18ed77e22c574781c\": container with ID starting with 219c9c04bfca53972ebe8b32b677f803bd87f967203339d18ed77e22c574781c not found: ID does not exist"
Sep 14 11:26:39.288693 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:39.288673    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-785cb4f769-52kcw"]
Sep 14 11:26:39.292979 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:39.292958    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-785cb4f769-52kcw"]
Sep 14 11:26:39.804021 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:39.803986    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd9e26e5-5c8f-4b62-bc93-0be0888e371f" path="/var/lib/kubelet/pods/cd9e26e5-5c8f-4b62-bc93-0be0888e371f/volumes"
Sep 14 11:26:40.062166 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.062081    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-785cb4f769-8b9gb"]
Sep 14 11:26:40.062482 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.062470    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd9e26e5-5c8f-4b62-bc93-0be0888e371f" containerName="manager"
Sep 14 11:26:40.062539 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.062484    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd9e26e5-5c8f-4b62-bc93-0be0888e371f" containerName="manager"
Sep 14 11:26:40.062539 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.062501    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy"
Sep 14 11:26:40.062539 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.062506    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f557605-f940-4ae6-9054-3dc0cee19d46" containerName="istio-proxy"
Sep 14 11:26:40.062539 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.062516    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="196e2ccf-eb11-4841-b102-7e600293c333" containerName="istio-proxy"
Sep 14 11:26:40.062539 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.062522    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="196e2ccf-eb11-4841-b102-7e600293c333" containerName="istio-proxy"
Sep 14 11:26:40.062729 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.062583    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd9e26e5-5c8f-4b62-bc93-0be0888e371f" containerName="manager"
Sep 14 11:26:40.066889 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.066871    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785cb4f769-8b9gb"
Sep 14 11:26:40.070682 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.070660    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lv5gj\""
Sep 14 11:26:40.070799 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.070683    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 14 11:26:40.070799 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.070695    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 14 11:26:40.070799 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.070662    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 14 11:26:40.077783 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.077762    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-785cb4f769-8b9gb"]
Sep 14 11:26:40.111032 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.111008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-295h5\" (UniqueName: \"kubernetes.io/projected/bbaeac8f-c715-4e67-9330-e59ce11467bc-kube-api-access-295h5\") pod \"maas-controller-785cb4f769-8b9gb\" (UID: \"bbaeac8f-c715-4e67-9330-e59ce11467bc\") " pod="opendatahub/maas-controller-785cb4f769-8b9gb"
Sep 14 11:26:40.111157 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.111137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbaeac8f-c715-4e67-9330-e59ce11467bc-webhook-cert\") pod \"maas-controller-785cb4f769-8b9gb\" (UID: \"bbaeac8f-c715-4e67-9330-e59ce11467bc\") " pod="opendatahub/maas-controller-785cb4f769-8b9gb"
Sep 14 11:26:40.111231 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.111191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbaeac8f-c715-4e67-9330-e59ce11467bc-metrics-certs\") pod \"maas-controller-785cb4f769-8b9gb\" (UID: \"bbaeac8f-c715-4e67-9330-e59ce11467bc\") " pod="opendatahub/maas-controller-785cb4f769-8b9gb"
Sep 14 11:26:40.211676 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.211652    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbaeac8f-c715-4e67-9330-e59ce11467bc-webhook-cert\") pod \"maas-controller-785cb4f769-8b9gb\" (UID: \"bbaeac8f-c715-4e67-9330-e59ce11467bc\") " pod="opendatahub/maas-controller-785cb4f769-8b9gb"
Sep 14 11:26:40.211842 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.211682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbaeac8f-c715-4e67-9330-e59ce11467bc-metrics-certs\") pod \"maas-controller-785cb4f769-8b9gb\" (UID: \"bbaeac8f-c715-4e67-9330-e59ce11467bc\") " pod="opendatahub/maas-controller-785cb4f769-8b9gb"
Sep 14 11:26:40.211842 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.211729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-295h5\" (UniqueName: \"kubernetes.io/projected/bbaeac8f-c715-4e67-9330-e59ce11467bc-kube-api-access-295h5\") pod \"maas-controller-785cb4f769-8b9gb\" (UID: \"bbaeac8f-c715-4e67-9330-e59ce11467bc\") " pod="opendatahub/maas-controller-785cb4f769-8b9gb"
Sep 14 11:26:40.214154 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.214126    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbaeac8f-c715-4e67-9330-e59ce11467bc-metrics-certs\") pod \"maas-controller-785cb4f769-8b9gb\" (UID: \"bbaeac8f-c715-4e67-9330-e59ce11467bc\") " pod="opendatahub/maas-controller-785cb4f769-8b9gb"
Sep 14 11:26:40.214248 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.214166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbaeac8f-c715-4e67-9330-e59ce11467bc-webhook-cert\") pod \"maas-controller-785cb4f769-8b9gb\" (UID: \"bbaeac8f-c715-4e67-9330-e59ce11467bc\") " pod="opendatahub/maas-controller-785cb4f769-8b9gb"
Sep 14 11:26:40.220198 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.220176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-295h5\" (UniqueName: \"kubernetes.io/projected/bbaeac8f-c715-4e67-9330-e59ce11467bc-kube-api-access-295h5\") pod \"maas-controller-785cb4f769-8b9gb\" (UID: \"bbaeac8f-c715-4e67-9330-e59ce11467bc\") " pod="opendatahub/maas-controller-785cb4f769-8b9gb"
Sep 14 11:26:40.378263 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.378237    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785cb4f769-8b9gb"
Sep 14 11:26:40.664886 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:40.664860    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-785cb4f769-8b9gb"]
Sep 14 11:26:40.666999 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:26:40.666969    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbaeac8f_c715_4e67_9330_e59ce11467bc.slice/crio-80fe818504f27e82fb704de24ed8288fdbf9f98e960a94f0626265eb3d30f767 WatchSource:0}: Error finding container 80fe818504f27e82fb704de24ed8288fdbf9f98e960a94f0626265eb3d30f767: Status 404 returned error can't find the container with id 80fe818504f27e82fb704de24ed8288fdbf9f98e960a94f0626265eb3d30f767
Sep 14 11:26:41.280762 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:41.280657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785cb4f769-8b9gb" event={"ID":"bbaeac8f-c715-4e67-9330-e59ce11467bc","Type":"ContainerStarted","Data":"e621652a08f6e64cbe74e73792c72bf02db2989477cd9f789a281e2d93fd1409"}
Sep 14 11:26:41.280762 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:41.280731    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785cb4f769-8b9gb" event={"ID":"bbaeac8f-c715-4e67-9330-e59ce11467bc","Type":"ContainerStarted","Data":"80fe818504f27e82fb704de24ed8288fdbf9f98e960a94f0626265eb3d30f767"}
Sep 14 11:26:41.280954 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:41.280777    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-785cb4f769-8b9gb"
Sep 14 11:26:41.307269 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:41.307213    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-785cb4f769-8b9gb" podStartSLOduration=0.957851938 podStartE2EDuration="1.307196311s" totalImagesPullingTime="349.344373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:26:41.305030797 +0000 UTC m=+2305.989715207" watchObservedRunningTime="2026-09-14 11:26:41.307196311 +0000 UTC m=+2305.991880701"
Sep 14 11:26:52.288327 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:26:52.288298    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-785cb4f769-8b9gb"
Sep 14 11:27:04.058517 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:04.058475    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:27:11.612729 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:11.612633    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:27:16.296395 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:16.296360    2561 scope.go:117] "RemoveContainer" containerID="c49ab3538e0514d9c8e2038795ca177f8f38ef38ba2a14267d997b05bd2d938b"
Sep 14 11:27:20.227917 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:20.227883    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:27:35.804642 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:35.804609    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:27:42.541275 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:42.541241    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:27:45.341023 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:45.340986    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"]
Sep 14 11:27:45.341431 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:45.341207    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4" podUID="87dc3b98-f8dc-41f8-aa25-9fd9f802fca5" containerName="manager" containerID="cri-o://bf0e6ea807af7f7048f9a52dba962d0d7f1a325ae93da0290fce740b7808502e" gracePeriod=10
Sep 14 11:27:46.646943 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:46.646908    2561 generic.go:356] "Generic (PLEG): container finished" podID="87dc3b98-f8dc-41f8-aa25-9fd9f802fca5" containerID="bf0e6ea807af7f7048f9a52dba962d0d7f1a325ae93da0290fce740b7808502e" exitCode=0
Sep 14 11:27:46.647259 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:46.646971    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4" event={"ID":"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5","Type":"ContainerDied","Data":"bf0e6ea807af7f7048f9a52dba962d0d7f1a325ae93da0290fce740b7808502e"}
Sep 14 11:27:46.682792 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:46.682769    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"
Sep 14 11:27:46.780303 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:46.780275    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dckfd\" (UniqueName: \"kubernetes.io/projected/87dc3b98-f8dc-41f8-aa25-9fd9f802fca5-kube-api-access-dckfd\") pod \"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5\" (UID: \"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5\") "
Sep 14 11:27:46.780469 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:46.780331    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87dc3b98-f8dc-41f8-aa25-9fd9f802fca5-extensions-socket-volume\") pod \"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5\" (UID: \"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5\") "
Sep 14 11:27:46.780761 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:46.780734    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87dc3b98-f8dc-41f8-aa25-9fd9f802fca5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "87dc3b98-f8dc-41f8-aa25-9fd9f802fca5" (UID: "87dc3b98-f8dc-41f8-aa25-9fd9f802fca5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:27:46.782391 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:46.782367    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87dc3b98-f8dc-41f8-aa25-9fd9f802fca5-kube-api-access-dckfd" (OuterVolumeSpecName: "kube-api-access-dckfd") pod "87dc3b98-f8dc-41f8-aa25-9fd9f802fca5" (UID: "87dc3b98-f8dc-41f8-aa25-9fd9f802fca5"). InnerVolumeSpecName "kube-api-access-dckfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:27:46.881367 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:46.881325    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dckfd\" (UniqueName: \"kubernetes.io/projected/87dc3b98-f8dc-41f8-aa25-9fd9f802fca5-kube-api-access-dckfd\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:27:46.881367 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:46.881366    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87dc3b98-f8dc-41f8-aa25-9fd9f802fca5-extensions-socket-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:27:47.653526 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:47.653487    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"
Sep 14 11:27:47.653526 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:47.653513    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4" event={"ID":"87dc3b98-f8dc-41f8-aa25-9fd9f802fca5","Type":"ContainerDied","Data":"1d1871026816ee8354f2032fb644b2102990b66ba9fc16a22b41b0acce4d4633"}
Sep 14 11:27:47.654000 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:47.653609    2561 scope.go:117] "RemoveContainer" containerID="bf0e6ea807af7f7048f9a52dba962d0d7f1a325ae93da0290fce740b7808502e"
Sep 14 11:27:47.678211 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:47.678179    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"]
Sep 14 11:27:47.684849 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:47.684826    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d5zx4"]
Sep 14 11:27:47.802594 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:27:47.802566    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87dc3b98-f8dc-41f8-aa25-9fd9f802fca5" path="/var/lib/kubelet/pods/87dc3b98-f8dc-41f8-aa25-9fd9f802fca5/volumes"
Sep 14 11:28:16.053377 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:16.053346    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 11:28:16.061095 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:16.061070    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 11:28:16.320949 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:16.320876    2561 scope.go:117] "RemoveContainer" containerID="bbf18ef2f38729a3d47e04c2ab5399f05ac614e1176442de0643c2e772ee148a"
Sep 14 11:28:46.036390 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.036313    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r"]
Sep 14 11:28:46.036855 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.036671    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87dc3b98-f8dc-41f8-aa25-9fd9f802fca5" containerName="manager"
Sep 14 11:28:46.036855 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.036682    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87dc3b98-f8dc-41f8-aa25-9fd9f802fca5" containerName="manager"
Sep 14 11:28:46.036855 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.036771    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87dc3b98-f8dc-41f8-aa25-9fd9f802fca5" containerName="manager"
Sep 14 11:28:46.039826 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.039802    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r"
Sep 14 11:28:46.043442 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.043427    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bh4bq\""
Sep 14 11:28:46.063064 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.063034    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r"]
Sep 14 11:28:46.178227 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.178199    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq2ws\" (UniqueName: \"kubernetes.io/projected/1ad06951-2a63-434f-a688-fda648910416-kube-api-access-gq2ws\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k546r\" (UID: \"1ad06951-2a63-434f-a688-fda648910416\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r"
Sep 14 11:28:46.178397 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.178235    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ad06951-2a63-434f-a688-fda648910416-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k546r\" (UID: \"1ad06951-2a63-434f-a688-fda648910416\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r"
Sep 14 11:28:46.278729 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.278686    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq2ws\" (UniqueName: \"kubernetes.io/projected/1ad06951-2a63-434f-a688-fda648910416-kube-api-access-gq2ws\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k546r\" (UID: \"1ad06951-2a63-434f-a688-fda648910416\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r"
Sep 14 11:28:46.278889 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.278750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ad06951-2a63-434f-a688-fda648910416-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k546r\" (UID: \"1ad06951-2a63-434f-a688-fda648910416\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r"
Sep 14 11:28:46.279114 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.279094    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ad06951-2a63-434f-a688-fda648910416-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k546r\" (UID: \"1ad06951-2a63-434f-a688-fda648910416\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r"
Sep 14 11:28:46.293026 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.292975    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq2ws\" (UniqueName: \"kubernetes.io/projected/1ad06951-2a63-434f-a688-fda648910416-kube-api-access-gq2ws\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k546r\" (UID: \"1ad06951-2a63-434f-a688-fda648910416\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r"
Sep 14 11:28:46.349981 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.349953    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r"
Sep 14 11:28:46.642416 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.642388    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r"]
Sep 14 11:28:46.644280 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:28:46.644252    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad06951_2a63_434f_a688_fda648910416.slice/crio-19ce09304007504f22a28eb337c97128482ef37424595f4885fd67cda19b05b7 WatchSource:0}: Error finding container 19ce09304007504f22a28eb337c97128482ef37424595f4885fd67cda19b05b7: Status 404 returned error can't find the container with id 19ce09304007504f22a28eb337c97128482ef37424595f4885fd67cda19b05b7
Sep 14 11:28:46.974521 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.974420    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r" event={"ID":"1ad06951-2a63-434f-a688-fda648910416","Type":"ContainerStarted","Data":"eb466ce06c464f0d0f3df4c524382b33aaa8b469c63216f0eefc8168c209582a"}
Sep 14 11:28:46.974521 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.974483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r" event={"ID":"1ad06951-2a63-434f-a688-fda648910416","Type":"ContainerStarted","Data":"19ce09304007504f22a28eb337c97128482ef37424595f4885fd67cda19b05b7"}
Sep 14 11:28:46.974780 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:46.974585    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r"
Sep 14 11:28:47.010571 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:47.010517    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r" podStartSLOduration=1.010501483 podStartE2EDuration="1.010501483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:28:47.008209475 +0000 UTC m=+2431.692893863" watchObservedRunningTime="2026-09-14 11:28:47.010501483 +0000 UTC m=+2431.695185872"
Sep 14 11:28:57.982993 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:28:57.982963    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k546r"
Sep 14 11:29:08.419517 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:29:08.419481    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:29:16.315163 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:29:16.315129    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:29:36.023601 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:29:36.023564    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:29:46.414188 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:29:46.414153    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:29:59.816346 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:29:59.816306    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:30:00.150833 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:00.150797    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823090-zn2t7"]
Sep 14 11:30:00.154658 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:00.154633    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-zn2t7"
Sep 14 11:30:00.159405 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:00.159380    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\""
Sep 14 11:30:00.165100 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:00.165081    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-zn2t7"
Sep 14 11:30:00.166697 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:00.166674    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823090-zn2t7"]
Sep 14 11:30:00.439728 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:00.439686    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823090-zn2t7"]
Sep 14 11:30:00.440776 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:30:00.440744    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode338f403_b107_47da_99fd_3f6960a0b891.slice/crio-e7630a8ff92d363bd08d86599b33e4ce8943dd8005af87c90b9c2feeb35ae087 WatchSource:0}: Error finding container e7630a8ff92d363bd08d86599b33e4ce8943dd8005af87c90b9c2feeb35ae087: Status 404 returned error can't find the container with id e7630a8ff92d363bd08d86599b33e4ce8943dd8005af87c90b9c2feeb35ae087
Sep 14 11:30:01.394819 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:01.394790    2561 generic.go:356] "Generic (PLEG): container finished" podID="e338f403-b107-47da-99fd-3f6960a0b891" containerID="aef2ed443040e0381a8e47d35fb4d44379110f09a17c2197cc887afe3b888826" exitCode=0
Sep 14 11:30:01.395178 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:01.394838    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-zn2t7" event={"ID":"e338f403-b107-47da-99fd-3f6960a0b891","Type":"ContainerDied","Data":"aef2ed443040e0381a8e47d35fb4d44379110f09a17c2197cc887afe3b888826"}
Sep 14 11:30:01.395178 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:01.394881    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-zn2t7" event={"ID":"e338f403-b107-47da-99fd-3f6960a0b891","Type":"ContainerStarted","Data":"e7630a8ff92d363bd08d86599b33e4ce8943dd8005af87c90b9c2feeb35ae087"}
Sep 14 11:30:02.539442 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:02.539414    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-zn2t7"
Sep 14 11:30:03.408502 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:03.408470    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-zn2t7"
Sep 14 11:30:03.408672 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:03.408497    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823090-zn2t7" event={"ID":"e338f403-b107-47da-99fd-3f6960a0b891","Type":"ContainerDied","Data":"e7630a8ff92d363bd08d86599b33e4ce8943dd8005af87c90b9c2feeb35ae087"}
Sep 14 11:30:03.408672 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:03.408543    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7630a8ff92d363bd08d86599b33e4ce8943dd8005af87c90b9c2feeb35ae087"
Sep 14 11:30:08.915702 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:08.915624    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:30:15.514873 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:15.514837    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:30:22.410277 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.410237    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:30:22.805135 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.805055    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"]
Sep 14 11:30:22.805532 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.805516    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e338f403-b107-47da-99fd-3f6960a0b891" containerName="cleanup"
Sep 14 11:30:22.805578 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.805534    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e338f403-b107-47da-99fd-3f6960a0b891" containerName="cleanup"
Sep 14 11:30:22.805622 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.805605    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e338f403-b107-47da-99fd-3f6960a0b891" containerName="cleanup"
Sep 14 11:30:22.808887 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.808871    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:22.812906 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.812877    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksb7s\""
Sep 14 11:30:22.813049 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.812915    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 11:30:22.813049 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.812922    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 11:30:22.813049 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.812885    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 14 11:30:22.813049 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.813028    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 11:30:22.820169 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.820147    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"]
Sep 14 11:30:22.826023 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.826002    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"]
Sep 14 11:30:22.829400 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.829382    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:22.845953 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.845932    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"]
Sep 14 11:30:22.919471 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.919429    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94pc4\" (UniqueName: \"kubernetes.io/projected/66bc4dc4-d639-4086-8035-83b7b7120bee-kube-api-access-94pc4\") pod \"maas-api-67455dc876-2tvm7\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") " pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:22.919651 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.919486    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66bc4dc4-d639-4086-8035-83b7b7120bee-metrics-certs\") pod \"maas-api-67455dc876-2tvm7\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") " pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:22.919651 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.919540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bc4dc4-d639-4086-8035-83b7b7120bee-openshift-service-ca\") pod \"maas-api-67455dc876-2tvm7\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") " pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:22.919651 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.919578    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5d6a3a-e8d9-4c56-bec0-da568f61a830-openshift-service-ca\") pod \"maas-api-6ffbf7464d-8z55t\" (UID: \"4e5d6a3a-e8d9-4c56-bec0-da568f61a830\") " pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:22.919651 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.919630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5d6a3a-e8d9-4c56-bec0-da568f61a830-maas-api-tls\") pod \"maas-api-6ffbf7464d-8z55t\" (UID: \"4e5d6a3a-e8d9-4c56-bec0-da568f61a830\") " pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:22.919901 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.919695    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5d6a3a-e8d9-4c56-bec0-da568f61a830-metrics-certs\") pod \"maas-api-6ffbf7464d-8z55t\" (UID: \"4e5d6a3a-e8d9-4c56-bec0-da568f61a830\") " pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:22.919901 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.919775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bc4dc4-d639-4086-8035-83b7b7120bee-maas-api-tls\") pod \"maas-api-67455dc876-2tvm7\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") " pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:22.919901 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:22.919822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t8cn\" (UniqueName: \"kubernetes.io/projected/4e5d6a3a-e8d9-4c56-bec0-da568f61a830-kube-api-access-2t8cn\") pod \"maas-api-6ffbf7464d-8z55t\" (UID: \"4e5d6a3a-e8d9-4c56-bec0-da568f61a830\") " pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:23.021049 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.021012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5d6a3a-e8d9-4c56-bec0-da568f61a830-openshift-service-ca\") pod \"maas-api-6ffbf7464d-8z55t\" (UID: \"4e5d6a3a-e8d9-4c56-bec0-da568f61a830\") " pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:23.021211 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.021077    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5d6a3a-e8d9-4c56-bec0-da568f61a830-maas-api-tls\") pod \"maas-api-6ffbf7464d-8z55t\" (UID: \"4e5d6a3a-e8d9-4c56-bec0-da568f61a830\") " pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:23.021211 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.021127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5d6a3a-e8d9-4c56-bec0-da568f61a830-metrics-certs\") pod \"maas-api-6ffbf7464d-8z55t\" (UID: \"4e5d6a3a-e8d9-4c56-bec0-da568f61a830\") " pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:23.021211 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.021164    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bc4dc4-d639-4086-8035-83b7b7120bee-maas-api-tls\") pod \"maas-api-67455dc876-2tvm7\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") " pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:23.021211 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.021193    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t8cn\" (UniqueName: \"kubernetes.io/projected/4e5d6a3a-e8d9-4c56-bec0-da568f61a830-kube-api-access-2t8cn\") pod \"maas-api-6ffbf7464d-8z55t\" (UID: \"4e5d6a3a-e8d9-4c56-bec0-da568f61a830\") " pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:23.021458 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.021261    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94pc4\" (UniqueName: \"kubernetes.io/projected/66bc4dc4-d639-4086-8035-83b7b7120bee-kube-api-access-94pc4\") pod \"maas-api-67455dc876-2tvm7\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") " pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:23.021458 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.021304    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66bc4dc4-d639-4086-8035-83b7b7120bee-metrics-certs\") pod \"maas-api-67455dc876-2tvm7\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") " pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:23.021458 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.021355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bc4dc4-d639-4086-8035-83b7b7120bee-openshift-service-ca\") pod \"maas-api-67455dc876-2tvm7\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") " pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:23.022074 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.021823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5d6a3a-e8d9-4c56-bec0-da568f61a830-openshift-service-ca\") pod \"maas-api-6ffbf7464d-8z55t\" (UID: \"4e5d6a3a-e8d9-4c56-bec0-da568f61a830\") " pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:23.022227 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.022102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bc4dc4-d639-4086-8035-83b7b7120bee-openshift-service-ca\") pod \"maas-api-67455dc876-2tvm7\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") " pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:23.024582 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.024552    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5d6a3a-e8d9-4c56-bec0-da568f61a830-metrics-certs\") pod \"maas-api-6ffbf7464d-8z55t\" (UID: \"4e5d6a3a-e8d9-4c56-bec0-da568f61a830\") " pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:23.024731 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.024607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5d6a3a-e8d9-4c56-bec0-da568f61a830-maas-api-tls\") pod \"maas-api-6ffbf7464d-8z55t\" (UID: \"4e5d6a3a-e8d9-4c56-bec0-da568f61a830\") " pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:23.025090 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.025065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bc4dc4-d639-4086-8035-83b7b7120bee-maas-api-tls\") pod \"maas-api-67455dc876-2tvm7\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") " pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:23.025200 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.025180    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66bc4dc4-d639-4086-8035-83b7b7120bee-metrics-certs\") pod \"maas-api-67455dc876-2tvm7\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") " pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:23.030688 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.030668    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94pc4\" (UniqueName: \"kubernetes.io/projected/66bc4dc4-d639-4086-8035-83b7b7120bee-kube-api-access-94pc4\") pod \"maas-api-67455dc876-2tvm7\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") " pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:23.030893 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.030875    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t8cn\" (UniqueName: \"kubernetes.io/projected/4e5d6a3a-e8d9-4c56-bec0-da568f61a830-kube-api-access-2t8cn\") pod \"maas-api-6ffbf7464d-8z55t\" (UID: \"4e5d6a3a-e8d9-4c56-bec0-da568f61a830\") " pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:23.120097 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.120061    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:23.140360 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.140331    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:23.399170 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.399145    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"]
Sep 14 11:30:23.399909 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:30:23.399887    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bc4dc4_d639_4086_8035_83b7b7120bee.slice/crio-8add36c8a8e2609351d420bf35d3fada33471d4f5b3a223e61a5b7048f1c0146 WatchSource:0}: Error finding container 8add36c8a8e2609351d420bf35d3fada33471d4f5b3a223e61a5b7048f1c0146: Status 404 returned error can't find the container with id 8add36c8a8e2609351d420bf35d3fada33471d4f5b3a223e61a5b7048f1c0146
Sep 14 11:30:23.429516 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.429491    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"]
Sep 14 11:30:23.430410 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:30:23.430386    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5d6a3a_e8d9_4c56_bec0_da568f61a830.slice/crio-04446dda2464963025b00af634a2a8dcce9facce79272ef59357c56c6fbda98b WatchSource:0}: Error finding container 04446dda2464963025b00af634a2a8dcce9facce79272ef59357c56c6fbda98b: Status 404 returned error can't find the container with id 04446dda2464963025b00af634a2a8dcce9facce79272ef59357c56c6fbda98b
Sep 14 11:30:23.520945 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.520904    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7" event={"ID":"66bc4dc4-d639-4086-8035-83b7b7120bee","Type":"ContainerStarted","Data":"8add36c8a8e2609351d420bf35d3fada33471d4f5b3a223e61a5b7048f1c0146"}
Sep 14 11:30:23.522058 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:23.522035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t" event={"ID":"4e5d6a3a-e8d9-4c56-bec0-da568f61a830","Type":"ContainerStarted","Data":"04446dda2464963025b00af634a2a8dcce9facce79272ef59357c56c6fbda98b"}
Sep 14 11:30:24.531188 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:24.531148    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t" event={"ID":"4e5d6a3a-e8d9-4c56-bec0-da568f61a830","Type":"ContainerStarted","Data":"aa606cfa61178d5ed8158745084677b82adb3140767647ce714b9e727bf5fcc5"}
Sep 14 11:30:24.531636 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:24.531206    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:24.533213 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:24.533188    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7" event={"ID":"66bc4dc4-d639-4086-8035-83b7b7120bee","Type":"ContainerStarted","Data":"35978f3535c4b695bd7a863cb4bffa697ad0c8472cc5f4564266991fd7e1e373"}
Sep 14 11:30:24.533442 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:24.533409    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:24.552777 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:24.552730    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t" podStartSLOduration=2.10023669 podStartE2EDuration="2.552702187s" totalImagesPullingTime="452.465497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:30:24.54996402 +0000 UTC m=+2529.234648412" watchObservedRunningTime="2026-09-14 11:30:24.552702187 +0000 UTC m=+2529.237386601"
Sep 14 11:30:24.573021 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:24.572974    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7" podStartSLOduration=2.172037377 podStartE2EDuration="2.572954946s" totalImagesPullingTime="400.917569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:30:24.570395192 +0000 UTC m=+2529.255079582" watchObservedRunningTime="2026-09-14 11:30:24.572954946 +0000 UTC m=+2529.257639334"
Sep 14 11:30:30.207470 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.207437    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:30:30.545848 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.545767    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:30.546262 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.546239    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6ffbf7464d-8z55t"
Sep 14 11:30:30.627358 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.627322    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"]
Sep 14 11:30:30.627566 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.627542    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7" podUID="66bc4dc4-d639-4086-8035-83b7b7120bee" containerName="maas-api" containerID="cri-o://35978f3535c4b695bd7a863cb4bffa697ad0c8472cc5f4564266991fd7e1e373" gracePeriod=30
Sep 14 11:30:30.907529 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.907503    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:30.991752 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.991701    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94pc4\" (UniqueName: \"kubernetes.io/projected/66bc4dc4-d639-4086-8035-83b7b7120bee-kube-api-access-94pc4\") pod \"66bc4dc4-d639-4086-8035-83b7b7120bee\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") "
Sep 14 11:30:30.991952 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.991760    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bc4dc4-d639-4086-8035-83b7b7120bee-maas-api-tls\") pod \"66bc4dc4-d639-4086-8035-83b7b7120bee\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") "
Sep 14 11:30:30.991952 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.991890    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bc4dc4-d639-4086-8035-83b7b7120bee-openshift-service-ca\") pod \"66bc4dc4-d639-4086-8035-83b7b7120bee\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") "
Sep 14 11:30:30.992082 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.991996    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66bc4dc4-d639-4086-8035-83b7b7120bee-metrics-certs\") pod \"66bc4dc4-d639-4086-8035-83b7b7120bee\" (UID: \"66bc4dc4-d639-4086-8035-83b7b7120bee\") "
Sep 14 11:30:30.992292 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.992268    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bc4dc4-d639-4086-8035-83b7b7120bee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66bc4dc4-d639-4086-8035-83b7b7120bee" (UID: "66bc4dc4-d639-4086-8035-83b7b7120bee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:30:30.994089 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.994053    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66bc4dc4-d639-4086-8035-83b7b7120bee-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "66bc4dc4-d639-4086-8035-83b7b7120bee" (UID: "66bc4dc4-d639-4086-8035-83b7b7120bee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:30:30.994089 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.994069    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bc4dc4-d639-4086-8035-83b7b7120bee-kube-api-access-94pc4" (OuterVolumeSpecName: "kube-api-access-94pc4") pod "66bc4dc4-d639-4086-8035-83b7b7120bee" (UID: "66bc4dc4-d639-4086-8035-83b7b7120bee"). InnerVolumeSpecName "kube-api-access-94pc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:30:30.994089 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:30.994076    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66bc4dc4-d639-4086-8035-83b7b7120bee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66bc4dc4-d639-4086-8035-83b7b7120bee" (UID: "66bc4dc4-d639-4086-8035-83b7b7120bee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:30:31.093742 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.093637    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94pc4\" (UniqueName: \"kubernetes.io/projected/66bc4dc4-d639-4086-8035-83b7b7120bee-kube-api-access-94pc4\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:30:31.093742 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.093669    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bc4dc4-d639-4086-8035-83b7b7120bee-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:30:31.093742 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.093678    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bc4dc4-d639-4086-8035-83b7b7120bee-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:30:31.093742 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.093687    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66bc4dc4-d639-4086-8035-83b7b7120bee-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\""
Sep 14 11:30:31.575617 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.575579    2561 generic.go:356] "Generic (PLEG): container finished" podID="66bc4dc4-d639-4086-8035-83b7b7120bee" containerID="35978f3535c4b695bd7a863cb4bffa697ad0c8472cc5f4564266991fd7e1e373" exitCode=0
Sep 14 11:30:31.576042 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.575629    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"
Sep 14 11:30:31.576042 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.575674    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7" event={"ID":"66bc4dc4-d639-4086-8035-83b7b7120bee","Type":"ContainerDied","Data":"35978f3535c4b695bd7a863cb4bffa697ad0c8472cc5f4564266991fd7e1e373"}
Sep 14 11:30:31.576042 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.575753    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67455dc876-2tvm7" event={"ID":"66bc4dc4-d639-4086-8035-83b7b7120bee","Type":"ContainerDied","Data":"8add36c8a8e2609351d420bf35d3fada33471d4f5b3a223e61a5b7048f1c0146"}
Sep 14 11:30:31.576042 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.575794    2561 scope.go:117] "RemoveContainer" containerID="35978f3535c4b695bd7a863cb4bffa697ad0c8472cc5f4564266991fd7e1e373"
Sep 14 11:30:31.585563 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.585547    2561 scope.go:117] "RemoveContainer" containerID="35978f3535c4b695bd7a863cb4bffa697ad0c8472cc5f4564266991fd7e1e373"
Sep 14 11:30:31.585851 ip-10-0-136-89 kubenswrapper[2561]: E0914 11:30:31.585830    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35978f3535c4b695bd7a863cb4bffa697ad0c8472cc5f4564266991fd7e1e373\": container with ID starting with 35978f3535c4b695bd7a863cb4bffa697ad0c8472cc5f4564266991fd7e1e373 not found: ID does not exist" containerID="35978f3535c4b695bd7a863cb4bffa697ad0c8472cc5f4564266991fd7e1e373"
Sep 14 11:30:31.585914 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.585859    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35978f3535c4b695bd7a863cb4bffa697ad0c8472cc5f4564266991fd7e1e373"} err="failed to get container status \"35978f3535c4b695bd7a863cb4bffa697ad0c8472cc5f4564266991fd7e1e373\": rpc error: code = NotFound desc = could not find container \"35978f3535c4b695bd7a863cb4bffa697ad0c8472cc5f4564266991fd7e1e373\": container with ID starting with 35978f3535c4b695bd7a863cb4bffa697ad0c8472cc5f4564266991fd7e1e373 not found: ID does not exist"
Sep 14 11:30:31.598481 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.598455    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"]
Sep 14 11:30:31.600874 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.600856    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-67455dc876-2tvm7"]
Sep 14 11:30:31.803372 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:31.803336    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66bc4dc4-d639-4086-8035-83b7b7120bee" path="/var/lib/kubelet/pods/66bc4dc4-d639-4086-8035-83b7b7120bee/volumes"
Sep 14 11:30:37.325982 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:37.325946    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:30:43.307918 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:43.307870    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:30:49.405634 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:49.405596    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-w6npf"]
Sep 14 11:30:57.960923 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:57.960891    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-785cb4f769-8b9gb_bbaeac8f-c715-4e67-9330-e59ce11467bc/manager/0.log"
Sep 14 11:30:58.067495 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:58.067469    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5l2v9_ab7827fb-8d35-490e-be09-cc313f86394d/server/0.log"
Sep 14 11:30:58.179221 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:58.179189    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hds9s_1c882741-8bd8-4818-bf4e-be21b6ba6599/manager/0.log"
Sep 14 11:30:58.289024 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:58.288957    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f6d976668-9bwzw_23430bd4-dfe5-440b-8f88-931054b06e46/copy-manifests/0.log"
Sep 14 11:30:58.295535 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:58.295516    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f6d976668-9bwzw_23430bd4-dfe5-440b-8f88-931054b06e46/manager/0.log"
Sep 14 11:30:58.963640 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:58.963612    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6ffbf7464d-8z55t_4e5d6a3a-e8d9-4c56-bec0-da568f61a830/maas-api/0.log"
Sep 14 11:30:59.175113 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:30:59.175084    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29823090-zn2t7_e338f403-b107-47da-99fd-3f6960a0b891/cleanup/0.log"
Sep 14 11:31:00.246902 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:00.246869    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-9cv9k_7ac50e28-8d32-43b5-80d2-4ae3ff80cb4d/manager/0.log"
Sep 14 11:31:00.471372 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:00.471337    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-k546r_1ad06951-2a63-434f-a688-fda648910416/manager/0.log"
Sep 14 11:31:00.589983 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:00.589906    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w6npf_1dd58c50-d680-4fe6-92cc-b2c8a3adbca3/limitador/0.log"
Sep 14 11:31:00.704177 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:00.704149    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-6kpts_89c873d9-7f29-4749-8896-3a2ce1e8f485/manager/0.log"
Sep 14 11:31:01.037525 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:01.037498    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr_b41ea28d-b80f-4039-9687-8e171a0cafee/istio-proxy/0.log"
Sep 14 11:31:08.121520 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.121481    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lp9t4/must-gather-sbp8j"]
Sep 14 11:31:08.122521 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.122487    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66bc4dc4-d639-4086-8035-83b7b7120bee" containerName="maas-api"
Sep 14 11:31:08.122521 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.122513    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="66bc4dc4-d639-4086-8035-83b7b7120bee" containerName="maas-api"
Sep 14 11:31:08.122749 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.122734    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="66bc4dc4-d639-4086-8035-83b7b7120bee" containerName="maas-api"
Sep 14 11:31:08.129589 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.129566    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp9t4/must-gather-sbp8j"
Sep 14 11:31:08.133638 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.133615    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lp9t4\"/\"default-dockercfg-4h7ts\""
Sep 14 11:31:08.133802 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.133615    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lp9t4\"/\"openshift-service-ca.crt\""
Sep 14 11:31:08.133802 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.133615    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lp9t4\"/\"kube-root-ca.crt\""
Sep 14 11:31:08.137006 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.136696    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp9t4/must-gather-sbp8j"]
Sep 14 11:31:08.221637 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.221609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa668841-60e8-4fde-8973-487e037a7d8a-must-gather-output\") pod \"must-gather-sbp8j\" (UID: \"aa668841-60e8-4fde-8973-487e037a7d8a\") " pod="openshift-must-gather-lp9t4/must-gather-sbp8j"
Sep 14 11:31:08.221791 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.221660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdbxj\" (UniqueName: \"kubernetes.io/projected/aa668841-60e8-4fde-8973-487e037a7d8a-kube-api-access-vdbxj\") pod \"must-gather-sbp8j\" (UID: \"aa668841-60e8-4fde-8973-487e037a7d8a\") " pod="openshift-must-gather-lp9t4/must-gather-sbp8j"
Sep 14 11:31:08.323086 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.323052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdbxj\" (UniqueName: \"kubernetes.io/projected/aa668841-60e8-4fde-8973-487e037a7d8a-kube-api-access-vdbxj\") pod \"must-gather-sbp8j\" (UID: \"aa668841-60e8-4fde-8973-487e037a7d8a\") " pod="openshift-must-gather-lp9t4/must-gather-sbp8j"
Sep 14 11:31:08.323217 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.323150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa668841-60e8-4fde-8973-487e037a7d8a-must-gather-output\") pod \"must-gather-sbp8j\" (UID: \"aa668841-60e8-4fde-8973-487e037a7d8a\") " pod="openshift-must-gather-lp9t4/must-gather-sbp8j"
Sep 14 11:31:08.323489 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.323471    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa668841-60e8-4fde-8973-487e037a7d8a-must-gather-output\") pod \"must-gather-sbp8j\" (UID: \"aa668841-60e8-4fde-8973-487e037a7d8a\") " pod="openshift-must-gather-lp9t4/must-gather-sbp8j"
Sep 14 11:31:08.331591 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.331568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdbxj\" (UniqueName: \"kubernetes.io/projected/aa668841-60e8-4fde-8973-487e037a7d8a-kube-api-access-vdbxj\") pod \"must-gather-sbp8j\" (UID: \"aa668841-60e8-4fde-8973-487e037a7d8a\") " pod="openshift-must-gather-lp9t4/must-gather-sbp8j"
Sep 14 11:31:08.439828 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.439747    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp9t4/must-gather-sbp8j"
Sep 14 11:31:08.714871 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.714841    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp9t4/must-gather-sbp8j"]
Sep 14 11:31:08.716366 ip-10-0-136-89 kubenswrapper[2561]: W0914 11:31:08.716337    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa668841_60e8_4fde_8973_487e037a7d8a.slice/crio-a53751ef2c765789cdf05df2eeaa18f81a9adf6a0309235db4f509e9cc3be1d3 WatchSource:0}: Error finding container a53751ef2c765789cdf05df2eeaa18f81a9adf6a0309235db4f509e9cc3be1d3: Status 404 returned error can't find the container with id a53751ef2c765789cdf05df2eeaa18f81a9adf6a0309235db4f509e9cc3be1d3
Sep 14 11:31:08.782150 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:08.782122    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp9t4/must-gather-sbp8j" event={"ID":"aa668841-60e8-4fde-8973-487e037a7d8a","Type":"ContainerStarted","Data":"a53751ef2c765789cdf05df2eeaa18f81a9adf6a0309235db4f509e9cc3be1d3"}
Sep 14 11:31:09.794210 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:09.793752    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp9t4/must-gather-sbp8j" event={"ID":"aa668841-60e8-4fde-8973-487e037a7d8a","Type":"ContainerStarted","Data":"63247126b814e15c80aa1a98dbb2d8265857d59361493626eefab46f8d360c5f"}
Sep 14 11:31:09.794210 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:09.793808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp9t4/must-gather-sbp8j" event={"ID":"aa668841-60e8-4fde-8973-487e037a7d8a","Type":"ContainerStarted","Data":"9ed0e0e7db3429dd07b33ce41bac2a1d8ea842e42a270200951985fb8f776120"}
Sep 14 11:31:09.813982 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:09.813914    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lp9t4/must-gather-sbp8j" podStartSLOduration=1.106171243 podStartE2EDuration="1.813893181s" totalImagesPullingTime="707.721938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:31:09.810705885 +0000 UTC m=+2574.495390275" watchObservedRunningTime="2026-09-14 11:31:09.813893181 +0000 UTC m=+2574.498577575"
Sep 14 11:31:11.054882 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:11.054839    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l5pvm_07dfb8b4-5bb4-4e2b-ab85-7b4dd7872029/global-pull-secret-syncer/0.log"
Sep 14 11:31:11.219952 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:11.219915    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z6bgb_c0aa6e02-ed99-41ac-82a1-7b9a7ab93993/konnectivity-agent/0.log"
Sep 14 11:31:11.242019 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:11.241991    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-89.ec2.internal_25265a84722357db880ad05e2abf2595/haproxy/0.log"
Sep 14 11:31:15.840429 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:15.840195    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-9cv9k_7ac50e28-8d32-43b5-80d2-4ae3ff80cb4d/manager/0.log"
Sep 14 11:31:15.915309 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:15.915263    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-k546r_1ad06951-2a63-434f-a688-fda648910416/manager/0.log"
Sep 14 11:31:15.938732 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:15.938687    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-w6npf_1dd58c50-d680-4fe6-92cc-b2c8a3adbca3/limitador/0.log"
Sep 14 11:31:16.019320 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:16.019283    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-6kpts_89c873d9-7f29-4749-8896-3a2ce1e8f485/manager/0.log"
Sep 14 11:31:17.520157 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:17.520125    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd05f595-b3de-48b3-b932-c29f72a7a287/alertmanager/0.log"
Sep 14 11:31:17.544842 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:17.544800    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd05f595-b3de-48b3-b932-c29f72a7a287/config-reloader/0.log"
Sep 14 11:31:17.566889 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:17.566853    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd05f595-b3de-48b3-b932-c29f72a7a287/kube-rbac-proxy-web/0.log"
Sep 14 11:31:17.587740 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:17.587699    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd05f595-b3de-48b3-b932-c29f72a7a287/kube-rbac-proxy/0.log"
Sep 14 11:31:17.607847 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:17.607788    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd05f595-b3de-48b3-b932-c29f72a7a287/kube-rbac-proxy-metric/0.log"
Sep 14 11:31:17.632804 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:17.632778    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd05f595-b3de-48b3-b932-c29f72a7a287/prom-label-proxy/0.log"
Sep 14 11:31:17.653479 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:17.653454    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd05f595-b3de-48b3-b932-c29f72a7a287/init-config-reloader/0.log"
Sep 14 11:31:17.720773 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:17.720744    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-vjsgs_35e5e4ed-da13-4019-9a71-045bff257926/kube-state-metrics/0.log"
Sep 14 11:31:17.738972 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:17.738944    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-vjsgs_35e5e4ed-da13-4019-9a71-045bff257926/kube-rbac-proxy-main/0.log"
Sep 14 11:31:17.767411 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:17.767382    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-vjsgs_35e5e4ed-da13-4019-9a71-045bff257926/kube-rbac-proxy-self/0.log"
Sep 14 11:31:17.818375 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:17.818310    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-5vz8d_0a8b5c7b-6176-47f7-b7f8-764efbf6f428/monitoring-plugin/0.log"
Sep 14 11:31:18.002036 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:18.002006    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hm8jf_db5b736c-6692-4802-9be1-2e6812fb2cd1/node-exporter/0.log"
Sep 14 11:31:18.033521 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:18.033484    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hm8jf_db5b736c-6692-4802-9be1-2e6812fb2cd1/kube-rbac-proxy/0.log"
Sep 14 11:31:18.057252 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:18.057225    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hm8jf_db5b736c-6692-4802-9be1-2e6812fb2cd1/init-textfile/0.log"
Sep 14 11:31:18.636935 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:18.636859    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-mr9bl_c871359e-4498-4328-977b-0ddbd9dd39a7/prometheus-operator-admission-webhook/0.log"
Sep 14 11:31:18.686372 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:18.686346    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-688f8d78d8-mrd7g_4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6/telemeter-client/0.log"
Sep 14 11:31:18.710159 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:18.710072    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-688f8d78d8-mrd7g_4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6/reload/0.log"
Sep 14 11:31:18.733166 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:18.733133    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-688f8d78d8-mrd7g_4f57ac5a-74bd-4ea4-9851-1ae3e9c4dcf6/kube-rbac-proxy/0.log"
Sep 14 11:31:20.993096 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:20.992994    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/1.log"
Sep 14 11:31:20.999578 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:20.999553    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-fxv4b_834e2d7c-b518-4988-a840-384226d6233d/console-operator/2.log"
Sep 14 11:31:21.055374 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.055343    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"]
Sep 14 11:31:21.061450 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.061424    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.067464 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.067427    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"]
Sep 14 11:31:21.175821 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.175784    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n2x4\" (UniqueName: \"kubernetes.io/projected/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-kube-api-access-4n2x4\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.176118 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.176101    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-lib-modules\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.176283 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.176266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-proc\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.176427 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.176415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-podres\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.176546 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.176534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-sys\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.277620 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.277530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-proc\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.277620 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.277604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-podres\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.277894 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.277644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-sys\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.277894 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.277687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-proc\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.277894 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.277752    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-sys\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.277894 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.277753    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-podres\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.277894 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.277860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n2x4\" (UniqueName: \"kubernetes.io/projected/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-kube-api-access-4n2x4\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.277894 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.277891    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-lib-modules\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.278194 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.278073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-lib-modules\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.289492 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.289467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n2x4\" (UniqueName: \"kubernetes.io/projected/44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf-kube-api-access-4n2x4\") pod \"perf-node-gather-daemonset-knpn5\" (UID: \"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.381813 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.381780    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.560416 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.560337    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66b78f9596-w8tbf_bfba9294-743c-4057-a30e-f51366280342/console/0.log"
Sep 14 11:31:21.603774 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.603743    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-2rk97_00887c1f-3e0f-4a86-8e54-35e02c9ffa53/download-server/0.log"
Sep 14 11:31:21.681015 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.680981    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"]
Sep 14 11:31:21.899959 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.899913    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5" event={"ID":"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf","Type":"ContainerStarted","Data":"2510168ee3b9585eb96f4c87cc7f742d0eecbbc1ee381fd2122c9246c0f3d822"}
Sep 14 11:31:21.900166 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.899986    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5" event={"ID":"44a4fae7-3c2c-48d4-9fb6-f9d858b7bcaf","Type":"ContainerStarted","Data":"35c2d27e441ae6d31c1e4f51a2b6117f58cf415c611b05fee3a2386ed266c2e7"}
Sep 14 11:31:21.900166 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.900029    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:21.920651 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:21.920594    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5" podStartSLOduration=0.920572547 podStartE2EDuration="920.572547ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:31:21.916933646 +0000 UTC m=+2586.601618037" watchObservedRunningTime="2026-09-14 11:31:21.920572547 +0000 UTC m=+2586.605256939"
Sep 14 11:31:23.333410 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:23.333369    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lbplh_90f711db-6849-4eb1-9ed2-85b410b8765e/dns/0.log"
Sep 14 11:31:23.361308 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:23.361280    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lbplh_90f711db-6849-4eb1-9ed2-85b410b8765e/kube-rbac-proxy/0.log"
Sep 14 11:31:23.452964 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:23.452937    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nr9n4_33c59f54-75fd-4277-86ad-b69ec8f49176/dns-node-resolver/0.log"
Sep 14 11:31:24.068468 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:24.068443    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xmbk7_914ecb92-cd00-4b96-bed0-d0df7d566386/node-ca/0.log"
Sep 14 11:31:24.979565 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:24.979530    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr_b41ea28d-b80f-4039-9687-8e171a0cafee/istio-proxy/0.log"
Sep 14 11:31:26.396916 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:26.396893    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m77xr_5b84cb0e-90da-4eee-85db-275a9795abca/serve-healthcheck-canary/0.log"
Sep 14 11:31:26.974063 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:26.974035    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ftztr_be8bbde4-d6c1-4267-bf03-c57928bc8e04/kube-rbac-proxy/0.log"
Sep 14 11:31:26.993704 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:26.993676    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ftztr_be8bbde4-d6c1-4267-bf03-c57928bc8e04/exporter/0.log"
Sep 14 11:31:27.013837 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:27.013806    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ftztr_be8bbde4-d6c1-4267-bf03-c57928bc8e04/extractor/0.log"
Sep 14 11:31:27.919142 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:27.919110    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-knpn5"
Sep 14 11:31:29.435894 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:29.435863    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-785cb4f769-8b9gb_bbaeac8f-c715-4e67-9330-e59ce11467bc/manager/0.log"
Sep 14 11:31:29.454943 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:29.454916    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5l2v9_ab7827fb-8d35-490e-be09-cc313f86394d/server/0.log"
Sep 14 11:31:29.576643 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:29.576577    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hds9s_1c882741-8bd8-4818-bf4e-be21b6ba6599/manager/0.log"
Sep 14 11:31:29.610823 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:29.610706    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f6d976668-9bwzw_23430bd4-dfe5-440b-8f88-931054b06e46/manager/0.log"
Sep 14 11:31:29.633401 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:29.633368    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f6d976668-9bwzw_23430bd4-dfe5-440b-8f88-931054b06e46/copy-manifests/0.log"
Sep 14 11:31:31.235944 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:31.235914    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-587bf9cb9-qsxsf_950120ed-49ef-4418-9409-860db1753d4b/manager/0.log"
Sep 14 11:31:31.271474 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:31.268766    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-xg5xh_6b8a38c7-6280-4985-8ab9-d8b527da060e/openshift-lws-operator/0.log"
Sep 14 11:31:31.819174 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:31.819144    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr_b41ea28d-b80f-4039-9687-8e171a0cafee/istio-proxy/0.log"
Sep 14 11:31:33.837581 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:33.837553    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-p7qjr_b41ea28d-b80f-4039-9687-8e171a0cafee/istio-proxy/0.log"
Sep 14 11:31:37.714093 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:37.714060    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4rjnr_20aafb29-e7c1-4ab2-94ab-a872003f7ae1/kube-multus/0.log"
Sep 14 11:31:38.039812 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:38.039693    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbbc4_6db78d45-a197-4361-9e83-fc3929395f94/kube-multus-additional-cni-plugins/0.log"
Sep 14 11:31:38.059634 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:38.059609    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbbc4_6db78d45-a197-4361-9e83-fc3929395f94/egress-router-binary-copy/0.log"
Sep 14 11:31:38.079689 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:38.079664    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbbc4_6db78d45-a197-4361-9e83-fc3929395f94/cni-plugins/0.log"
Sep 14 11:31:38.098736 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:38.098699    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbbc4_6db78d45-a197-4361-9e83-fc3929395f94/bond-cni-plugin/0.log"
Sep 14 11:31:38.118263 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:38.118236    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbbc4_6db78d45-a197-4361-9e83-fc3929395f94/routeoverride-cni/0.log"
Sep 14 11:31:38.138566 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:38.138541    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbbc4_6db78d45-a197-4361-9e83-fc3929395f94/whereabouts-cni-bincopy/0.log"
Sep 14 11:31:38.157468 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:38.157436    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbbc4_6db78d45-a197-4361-9e83-fc3929395f94/whereabouts-cni/0.log"
Sep 14 11:31:38.303125 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:38.303047    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-67w9c_b3d78350-bf09-4796-9e47-d39ec74fde78/network-metrics-daemon/0.log"
Sep 14 11:31:38.319233 ip-10-0-136-89 kubenswrapper[2561]: I0914 11:31:38.319211    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-67w9c_b3d78350-bf09-4796-9e47-d39ec74fde78/kube-rbac-proxy/0.log"